Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline

Abstract

We study task-agnostic continual reinforcement learning (TACRL) in which standard RL challenges are compounded with partial observability stemming from task agnosticism, as well as additional difficulties of continual learning (CL), i.e., learning on a non-stationary sequence of tasks. Here we compare TACRL methods with their soft upper bounds prescribed by previous literature: multi-task learning (MTL) methods which do not have to deal with non-stationary data distributions, as well as task-aware methods, which are allowed to operate under full observability. We consider a previously unexplored and straightforward baseline for TACRL, replay-based recurrent RL (3RL), in which we augment an RL algorithm with recurrent mechanisms to address partial observability and experience replay mechanisms to address catastrophic forgetting in CL. Studying empirical performance in a sequence of RL tasks, we find surprising occurrences of 3RL matching and overcoming the MTL and task-aware soft upper bounds. We lay out hypotheses that could explain this inflection point of continual and task-agnostic learning research. Our hypotheses are empirically tested in continuous control tasks via a large-scale study of the popular multi-task and continual learning benchmark Meta-World. By analyzing different training statistics including gradient conflict, we find evidence that 3RL's outperformance stems from its ability to quickly infer how new tasks relate with the previous ones, enabling forward transfer.

Learning Vision-based Robotic Manipulation Tasks Sequentially in Offline Reinforcement Learning Settings

“…On the other hand, our work is about learning tasks sequentially with only the current task data available for learning. Some very recent works try to apply continual RL on robotics manipulation tasks [22], [4]. [22] introduces a continual learning benchmark for robotic manipulation tasks.…”

Tutorial - Continual Learning beyond classification

“…It is important to note that those ratings classify the complexity of a given scenario. For example, the default scenario (classincremental) assesses a complexity level of (0 1), domain incremental without task labels would be (0 2), task agnostic continual RL [47] (1 3). To validate the autonomy of an approach, it should be evaluated in adequate scenarios.…”

“…A variety of approaches has been published, among which knowledge-based distillations [27,28] and context-based decompositions [105,106] are popular. Other works are concerned with the employed model [107,108,109,110], off-policy algorithms [111], policy gradient [112] or a task-agnostic perspective [47]. Evaluations of known CL methods (e.g., GEM, A-GEM, and replay) are also applied in the RL domain [113,9].…”