Abstract-In this paper, optimal critic learning is developed for robot control in a time-varying environment. The unknown environment is described as a linear system with time-varying parameters, and impedance control is employed for the interaction control. Desired impedance parameters are obtained in the sense of an optimal realization of the composite of trajectory tracking and force regulation. Q-function based critic learning is developed to determine the optimal impedance parameters without the knowledge of the system dynamics. Simulation results are presented and compared with existing methods, and the efficacy of the proposed method is verified.