A Multi-Target Trajectory Planning of a 6-DoF Free-Floating Space Robot via Reinforcement Learning

Abstract

Temporal difference (TD) learning is a fundamental technique in reinforcement learning that updates value estimates for states or state-action pairs using a TD target. This target represents an improved estimate of the true value by incorporating both immediate rewards and the estimated value of subsequent states. Traditionally, TD learning relies on the value of a single subsequent state. We propose an enhanced multi-state TD (MSTD) target that utilizes the estimated values of multiple subsequent states. Building on this new MSTD concept, we develop complete actor-critic algorithms that include management of replay buffers in two modes, and integrate with deep deterministic policy optimization (DDPG) and soft actorcritic (SAC). Experimental results demonstrate that algorithms employing the MSTD target significantly improve learning performance compared to traditional methods.

A Learning System for Motion Planning of Free-Float Dual-Arm Space Manipulator towards Non-Cooperative Object

“…For the free-floating space manipulator, Yan et al proposed a trajectory planning method based on Soft Q-learning for a 3-DoF free-floating space robot [17]. To reach multiple targets within a large space, Wang et al developed an improved version of Proximal Policy Optimization (PPO) for a 6-DoF space robot [25]. However, in our experiments, the method can not work well in a 12-DoF dual-arm environment.…”

“…Then a velocity tracking PD controller takes a t as input, to generate the torques of joints. Given that position controller performs less smoothly, it is better to choose the velocity controller [25].…”

“…• Wang's method [25]: Wang's method solved the task of multiple targets for a single arm based on an improved version of PPO algorithm.…”

“…• SAC-D: Considering Soft Actor-Critic (SAC) algorithm achieves superior performance in many robotic control task [37], we designed SAC-D algorithm for our task, in which we applied the similar reward function in [25].…”

“…The number of hidden units per layer is 256, and the activation of output layer in policy network is Tanh function. Additionally, the input and output are also the same variables, and other parameters in each algorithm are provided by some papers [18,25] or OpenAI baselines [38].…”