Learning modular neural network policies for multi-task and multi-robot transfer

Abstract

Reinforcement learning (RL) can automate a wide variety of robotic skills, but learning each new skill requires considerable real-world data collection and manual representation engineering to design policy classes or features. Using deep reinforcement learning to train general purpose neural network policies alleviates some of the burden of manual representation engineering by using expressive policy classes, but exacerbates the challenge of data collection, since such methods tend to be less efficient than RL with low-dimensional, hand-designed representations. Transfer learning can mitigate this problem by enabling us to transfer information from one skill to another and even from one robot to another. We show that neural network policies can be decomposed into "task-specific" and "robot-specific" modules, where the task-specific modules are shared across robots, and the robot-specific modules are shared across all tasks on that robot. This allows for sharing task information, such as perception, between robots and sharing robot information, such as dynamics and kinematics, between tasks. We exploit this decomposition to train mix-and-match modules that can solve new robot-task combinations that were not seen during training. Using a novel neural network architecture, we demonstrate the effectiveness of our transfer method for enabling zero-shot generalization with a variety of robots and tasks in simulation for both visual and non-visual tasks. * The first two authors contributed equally to this work.

Deep photovoltaic nowcasting

“…In addition, the auxiliary tasks in table 1-such as the sun position, the sky image, the sun position variation and the sky intensity variation-are independent from the solar panel. Evidence from the literature (Devin et al, 2017), which demonstrates that networks trained on multiple tasks have can better generalize on other datasets, leads us to believe that the "LSTM-Full" model may have a better chance of adapting to another site, for example with the same fine-tuning strategy mentioned above.…”

Deep Drone Racing: From Simulation to Reality With Domain Randomization

“…To alleviate some of these problems while retaining the advantages of data-driven methods, a number of works propose to structure the navigation system into two modules: perception and control [28]- [32]. This kind of modularity has proven to be particularly important for transferring sensorimotor systems across different tasks [29], [31] and application domains [30], [32].…”

Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

“…This is in contrast to previous works using unstructured task representations and policies [8,11]. The use of compositionality has led to better generalization in Visual Question Answering [17,20,24] and Policy Learning [3,7,38]. We propose Neural Task Graph (NTG) Networks, a novel framework that uses task graph as the intermediate representation to explicitly modularize both the visual demonstration and the derived policy.…”