Multi-Task Deep Reinforcement Learning with PopArt

Abstract

The reinforcement learning community has made great strides in designing algorithms capable of exceeding human performance on specific tasks. These algorithms are mostly trained one task at the time, each new task requiring to train a brand new agent instance. This means the learning algorithm is general, but each solution is not; each agent can only solve the one task it was trained on. In this work, we study the problem of learning to master not one but multiple sequentialdecision tasks at once. A general issue in multi-task learning is that a balance must be found between the needs of multiple tasks competing for the limited resources of a single learning system. Many learning algorithms can get distracted by certain tasks in the set of tasks to solve. Such tasks appear more salient to the learning process, for instance because of the density or magnitude of the in-task rewards. This causes the algorithm to focus on those salient tasks at the expense of generality. We propose to automatically adapt the contribution of each task to the agent's updates, so that all tasks have a similar impact on the learning dynamics. This resulted in state of the art performance on learning to play all games in a set of 57 diverse Atari games. Excitingly, our method learned a single trained policy -with a single set of weights -that exceeds median human performance. To our knowledge, this was the first time a single agent surpassed human-level performance on this multi-task domain. The same approach also demonstrated state of the art performance on a set of 30 tasks in the 3D reinforcement learning platform DeepMind Lab.

Simultaneous exploration and segmentation for search and rescue

“…We compared the following control policies: RGBD uses only the segmentation from and thus no thermal measurements. It provides a loose lower bound on the performance since the additional thermal modality provides an important cue with respect to the segmentation task and improves the performance in general, no matter what views are selected. DQN provides reactive control similar to Mnih et al () with the double DQN extension from Hasselt, Guez, and Silver () and the prioritized experience replay from Schaul, Quan, Antonoglou, and Silver (). Greedy corresponds to the network predicting the gain obtained through self‐supervision. The predicted pixel‐wise gain is accumulated by viewpoint kernels and the maximum within the motion constraints is selected for the next action. corresponds to the network obtained from the self‐supervised policy initialization. corresponds to the network fine‐tuned on the guiding trajectories () with previously trained to predict true gain . Optimal uses additional information of true to plan the optimal trajectory by solving instances of MILP.…”

“…We compared the following control policies: • DQN provides reactive control similar to Mnih et al (2015) with the double DQN extension from Hasselt, Guez, and Silver (2016) and the prioritized experience replay from Schaul, Quan, Antonoglou, and Silver (2016).…”

Path Planning for UAV-Mounted Mobile Edge Computing With Deep Reinforcement Learning

“…The concept of Q(s t , a t ) is to evaluate how good the action a t performed by the UAV in the state s t is. As illustrated in [14], DQN approximates the Q-value by using two deep neural networks (DNNs) with the same four fully connected layers but different parameters φ 1 and φ 2 . One is the predicted network, whose input is the current state-action pair (s t , a t ) and output is the predicted value, i.e., Q DQN predicted (s t , a t ; φ 1 ).…”

“…DQN structure chooses max a ′ Q(s t+1 , a ′ ; φ 2 ) directly in the target network, whose parameter is not updated timely and may lead to the overestimation of Q-value [14]. To address the overestimation problem, DDQN applies two independent estimators to approximate the Q-value.…”

“…Moreover, [13] proposed the deterministic policy gradient algorithm to maximize the expected uplink sum rate in the UAV-aided cellular networks with mobile TUs. Among the value based DRL algorithms, [14] unveiled that double deep Q-network (DDQN) addresses the overestimation problem in deep Q-network (DQN) via decoupling target Q-value and predicted Q-value, and generates a more accurate state-action value function than DQN. It is known that the better state-action value function corresponds to the better policy.…”