On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning

Abstract

Intelligent agents should have the ability to leverage knowledge from previously learned tasks in order to learn new ones quickly and efficiently. Meta-learning approaches have emerged as a popular solution to achieve this. However, metareinforcement learning (meta-RL) algorithms have thus far been restricted to simple environments with narrow task distributions. Moreover, the paradigm of pretraining followed by fine-tuning to adapt to new tasks has emerged as a simple yet effective solution in supervised and self-supervised learning. This calls into question the benefits of meta-learning approaches in reinforcement learning, which typically come at the cost of high complexity. We therefore investigate meta-RL approaches in a variety of vision-based benchmarks, including Procgen, RLBench, and Atari, where evaluations are made on completely novel tasks. Our findings show that when meta-learning approaches are evaluated on different tasks (rather than different variations of the same task), multi-task pretraining with fine-tuning on new tasks performs equally as well, or better, than meta-pretraining with meta test-time adaptation. This is encouraging for future research, as multi-task pretraining tends to be simpler and computationally cheaper than meta-RL. From these findings, we advocate for evaluating future meta-RL methods on more challenging tasks and including multi-task pretraining with fine-tuning as a simple, yet strong baseline. 1

Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning

“…As explained at the beginning of the paper, previous work has shown that using hardware data to fine-tune a policy that has been pre-trained in simulation is a powerful approach to tackle the sim-2-real gap problem (e.g. [12,13,14,15]). These methods typically take the RL agent trained in simulation and continue its learning process using hardware data.…”

“…Compared to these works, our primary contribution is to demonstrate how CLFs can be combined with model-free algorithms to rapidly learn stabilizing controllers for robotic systems. Fine-tuning with Real World Data: Recently, there has been much interest in using RL to finetune policies which have been pre-trained in simulation [12,13,14,15]. These methods typically optimize the same cost function with a large discount factor in both simulation and on the real robot.…”

“…However, despite these successes, there has been a renewed effort to incorporate data from hardware experiments into RL pipelines [11]. These two approaches are not at odds, as 'fine-tuning' [12,13,14,15] approaches can be used to improve a policy initially trained on a simulation model. This is an ideal middle ground which enables engineers to incorporate known features of the system into the model while still allowing model-free methods to improve performance by leveraging complex unknown nonlinearities in the environment.…”

Aligning Robot and Human Representations

“…The hope is that this representation holds meaningful structure for what's important for all tasks, thus letting the robot reuse the representation to efficiently learn new but related tasks. This has been shown to be more stable and scalable than metalearning [111], but still needs curating a large set of training tasks to robustly cover the test distribution.…”

Deep reinforcement learning for real-world quadrupedal locomotion: a comprehensive review

“…However, current meta-reinforcement learning algorithms are limited to simple environments with narrow task distributions [93][94][95][96] . A recent study showed that multi-task pre-training with fine-tuning on new tasks performs as well as or better than meta-pre-training with meta test-time adaptation [97] . Research considering large-scale pre-trained models in quadrupedal locomotion research is still in its infancy and needs further exploration.…”