Optimal Critic Learning for Robot Control in Time-Varying Environments

Abstract

Abstract-In this paper, optimal critic learning is developed for robot control in a time-varying environment. The unknown environment is described as a linear system with time-varying parameters, and impedance control is employed for the interaction control. Desired impedance parameters are obtained in the sense of an optimal realization of the composite of trajectory tracking and force regulation. Q-function based critic learning is developed to determine the optimal impedance parameters without the knowledge of the system dynamics. Simulation results are presented and compared with existing methods, and the efficacy of the proposed method is verified.

Adaptive impedance control of robot manipulators based on Q-learning and disturbance observer

“…In the model, the contact parameters relate the end-effector position x to the interaction force f e at each contact effector, C e and K e are unknown timevarying damping and stiffness matrices of the dynamics, respectively. Introducing an environment model proposed (Wang et al, 2015), we define that k describes the time-step index, the unknown time-varying environment dynamics in discrete time is given as follows:…”

“…Consider our previous results in Wang et al (2015) and the cost-to-go function in (17), a Q-function with quadratic form is introduced as follows:…”

“…In this subsection, we will employ a successive Q-learning method to solve (10) to obtain the sequence matrix P(k) (Wang et al, 2015), and impedance parameters are obtained by applying the Q-learning method. The algorithm is summarized as follows:…”

“…In our previous work Ge, Li, and Wang (2014), the developed method is verified for the time-invariant environment dynamics, such that the method is inapplicable to the time-varying environment interacted with the end-effector of robot manipulator. In Wang, Li, Ge, and Lee (2015), the optimal critic learning is proposed for unknown and time-varying environment, however, the uncertain effect of robot manipulator for trajectory tracking is not considered.…”

Position/force control of robot manipulators using reinforcement learning

“…When the environment is unknown, impedance control does not work well for interaction performance (Wang et al , 2015; Wen and Murphy, 1991; Wilfinger et al , 1994), and the force control is not robust to generate the desired strength with respect to uncertainties in the environment (Singh and Popa, 1995; Perrusquía et al , 2017). There are several methods to estimate the dynamics of the environment (Kang et al , 2009), such as adaptive identification (Ficuciello et al , 2015), recursive least squares (RLS) (Kelly et al , 1989), neural networks (Yamamoto et al , 2008) and the adaptive impedance algorithm (Lu and Meng, 1991).…”