Multisource Transfer Double DQN Based on Actor Learning

Abstract

Deep reinforcement learning (RL) comprehensively uses the psychological mechanisms of "trial and error" and "reward and punishment" in RL as well as powerful feature expression and nonlinear mapping in deep learning. Currently, it plays an essential role in the fields of artificial intelligence and machine learning. Since an RL agent needs to constantly interact with its surroundings, the deep Q network (DQN) is inevitably faced with the need to learn numerous network parameters, which results in low learning efficiency. In this paper, a multisource transfer double DQN (MTDDQN) based on actor learning is proposed. The transfer learning technique is integrated with deep RL to make the RL agent collect, summarize, and transfer action knowledge, including policy mimic and feature regression, to the training of related tasks. There exists action overestimation in DQN, i.e., the lower probability limit of action corresponding to the maximum Q value is nonzero. Therefore, the transfer network is trained by using double DQN to eliminate the error accumulation caused by action overestimation. In addition, to avoid negative transfer, i.e., to ensure strong correlations between source and target tasks, a multisource transfer learning mechanism is applied. The Atari2600 game is tested on the arcade learning environment platform to evaluate the feasibility and performance of MTDDQN by comparing it with some mainstream approaches, such as DQN and double DQN. Experiments prove that MTDDQN achieves not only human-like actor learning transfer capability, but also the desired learning efficiency and testing accuracy on target task.

Research on Dynamic Path Planning of Wheeled Robot Based on Deep Reinforcement Learning on the Slope Ground

“…In this section, the network structure consisting of three convolutional layers and one fully connected layer is proposed [16]. e CNN framework is shown in Figure 4.…”

Real-Time Optimal Power Flow Using Twin Delayed Deep Deterministic Policy Gradient Algorithm

“…If the overestimation continues to occur during training, the policy update will be negatively affected. These features have led to the emergence of techniques called Double Q-Learning and Double DQN, which use two value networks to separate action selection updates and Q-value updates [23]. TD3 (Twin Delayed Deep Deterministic Policy Gradient) is an algorithm that applies several techniques to the DDPG for preventing overestimation on the value function.…”

An intelligent course keeping active disturbance rejection controller based on double deep Q‐network for towing system of unpowered cylindrical drilling platform

“…However, DQN would overestimate the action-state value function during the training process, which affects the final decision and fails to obtain the optimal strategy. To solve this problem, many scholars put forward improved algorithms such as double DQN, 30 dueling DQN, 31 and rainbow DQN. 32 As far as we know, the double DQN plays a role in tuning optimization of the parameters for LADRC in this article for the first time.…”