A Survey of Zero-shot Generalisation in Deep Reinforcement Learning

Abstract

The study of generalisation in deep Reinforcement Learning (RL) aims to produce RL algorithms whose policies generalise well to novel unseen situations at deployment time, avoiding overfitting to their training environments. Tackling this is vital if we are to deploy reinforcement learning algorithms in real world scenarios, where the environment will be diverse, dynamic and unpredictable. This survey is an overview of this nascent field. We provide a unifying formalism and terminology for discussing different generalisation problems, building upon previous works. We go on to categorise existing benchmarks for generalisation, as well as current methods for tackling the generalisation problem. Finally, we provide a critical discussion of the current state of the field, including recommendations for future work. Among other conclusions, we argue that taking a purely procedural content generation approach to benchmark design is not conducive to progress in generalisation, we suggest fast online adaptation and tackling RL-specific problems as some areas for future work on methods for generalisation, and we recommend building benchmarks in underexplored problem settings such as offline RL generalisation and reward-function variation.

A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective

“…For instance, gradually training AVs with increasing risk level under curriculum learning [194] framework may help systems easily generalize to more types of safety-critical scenarios. One recent survey [195] that investigates the generalization problem in RL emphasizes the importance of environment generation in increasing the similarity between training and testing domains. This direction extends the scenario generation from safety to broader views that requires goalconditioned environment generation.…”

Robust Algorithmic Collusion

“…In order to distinguish different environments, we follow the ideas in Kirk et al (2021) and consider a Contextual Partially Observable Markov Game, where we introduce a set of contexts K. 5 For each context k ∈ K we have a Partially Observable Markov Game (POMG) with the property that the state of the game can be decomposed into two parts, s = (k, s ) ∈ S k , where s ∈ S is the state and k ∈ K is the context. Formally: Definition 2.2 (Contextual Partially Observable Markov Game (CPOMG)).…”

“…The learners then engage in a "meta-learning" problem with awareness of the contexts. This is the background of the framework introduced in Kirk et al (2021). Similarly, we will consider a discrete set of contexts in our analysis here.…”

“…Treutlein et al, 2021;Hu et al, 2020). Our framework builds on Kirk et al (2021). Related to our approach is a strand of literature that assumes there exists a distribution of Markov-decision-problems of the scenario of interest, and then trains algorithms on a finite set of samples from this distribution before testing the behavior on the entire distribution (e.g.…”

“…In the economic literature, these games are typically referred to as Stochastic Games. Here, we follow the terminology that is standard in the machine learning literature.5 Note thatKirk et al (2021) focus on a single agent and not interactions of players. Their formalization is therefore based on Partially Observable Markov Decision Problems.…”