Towards continual reinforcement learning through evolutionary meta-learning

Grbic, Djordje; Risi, Sebastian

doi:10.1145/3319619.3322044

Cited by 6 publications

(6 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach was first introduced by Finn et al (2017) and is called Model-Agnostic Meta-Learning (MAML). In the approach presented in this paper, we use an evolutionary meta-learning variant, in which evolution is trying to find good initial neural network parameters that allow an inner RL loop to adapt quickly (Fernando et al, 2018;Grbic and Risi, 2019).…”

Section: Meta-learningmentioning

confidence: 99%

“…The question here is how to train an instinctual network that keeps the agent out of harm's way together with a policy network that should be able to adapt quickly to new goals. One of the main insights in the work presented here is that we can use an evolutionary meta-learning approach (Fernando et al, 2018;Grbic and Risi, 2019) to train a policy that can adapt quickly and safely to different tasks. The whole training procedure runs two training loops: an evolutionary outer loop, and a task-adaptation inner loop (Alg.1 and Fig.…”

Section: Meta-trainingmentioning

confidence: 99%

“…The main idea in the approach introduced in this paper is to allow agents to evolve similar innate capabilities that help them to avoid potentially dangerous situations. The novel approach, called Meta-Learned Instinctual Networks (MLIN), builds on ideas from training agents for fast adaptation through meta-learning (Finn et al, 2017;Fernando et al, 2018;Grbic and Risi, 2019). A novel insight in this paper is that meta-learning can be an effective approach for AI safety by jointly evolving the initial parameters of a policy network that can adapt quickly during deployment through RL (Fernando et al, 2018), with the weights of an instinctual network that only changes during evolution and can modulate the noisy actions of the policy network to prevent the agent from encountering potentially dangerous situations (Fig.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Safe Reinforcement Learning through Meta-learned Instincts

Grbic¹,

Risi²

2020

The 2020 Conference on Artificial Life

Self Cite

View full text Add to dashboard Cite

An important goal in reinforcement learning is to create agents that can quickly adapt to new goals while avoiding situations that might cause damage to themselves or their environments. One way agents learn is through exploration mechanisms, which are needed to discover new policies. However, in deep reinforcement learning, exploration is normally done by injecting noise in the action space. While performing well in many domains, this setup has the inherent risk that the noisy actions performed by the agent lead to unsafe states in the environment. Here we introduce a novel approach called Meta-Learned Instinctual Networks (MLIN) that allows agents to safely learn during their lifetime while avoiding potentially hazardous states. At the core of the approach is a plastic network trained through reinforcement learning and an evolved "instinctual" network, which does not change during the agent's lifetime but can modulate the noisy output of the plastic network. We test our idea on a simple 2D navigation task with no-go zones, in which the agent has to learn to approach new targets during deployment. MLIN outperforms standard meta-trained networks and allows agents to learn to navigate to new targets without colliding with any of the no-go zones. These results suggest that meta-learning augmented with an instinctual network is a promising new approach for RL in safety-critical domains.

show abstract

Section: Meta-learningmentioning

confidence: 99%

Section: Meta-trainingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Safe Reinforcement Learning through Meta-learned Instincts

Grbic¹,

Risi²

2020

The 2020 Conference on Artificial Life

Self Cite

View full text Add to dashboard Cite

show abstract

“…One popular approach to address this problem is fewshot learning, in particular metalearning, either by utilizing gradients (Schmidhuber 1987;Thrun and Pratt 1998;Finn, Abbeel, and Levine 2017) or evolutionary procedures (Fernando et al 2018;Grbic and Risi 2019). In metalearning, systems are trained by exposing them to a large number of tasks, and then tested for their ability to learn relevant but previously unseen tasks.…”

Section: Introductionmentioning

confidence: 99%

Generalization of Agent Behavior through Explicit Representation of Context

Tutum¹,

Abdulquddos²,

Miikkulainen³

2020

Preprint

View full text Add to dashboard Cite

In order to deploy autonomous agents in digital interactive environments, they must be able to act robustly in unseen situations. The standard machine learning approach is to include as much variation as possible into training these agents. The agents can then interpolate within their training, but they cannot extrapolate much beyond it. This paper proposes a principled approach where a context module is coevolved with a skill module in the game. The context module recognizes the temporal variation in the game and modulates the outputs of the skill module so that the action decisions can be made robustly even in previously unseen situations. The approach is evaluated in the Flappy Bird and LunarLander video games, as well as in the CARLA autonomous driving simulation. The Context+Skill approach leads to significantly more robust behavior in environments that require extrapolation beyond training. Such a principled generalization ability is essential in deploying autonomous agents in real-world tasks, and can serve as a foundation for continual adaptation as well.

show abstract

“…One popular approach to address this problem is few-shot learning, in particular metalearning, either by utilizing gradients [7,21,25] or evolutionary procedures [6,10]. In metalearning, systems are trained by exposing them to a large number of tasks, and then tested for their ability to learn new relevant but unseen tasks.…”

Section: Introductionmentioning

confidence: 99%

Adapting to Unseen Environments through Explicit Representation of Context

Tutum¹,

Miikkulainen²

2020

Preprint

View full text Add to dashboard Cite

In order to deploy autonomous agents to domains such as autonomous driving, infrastructure management, health care, and finance, they must be able to adapt safely to unseen situations. The current approach in constructing such agents is to try to include as much variation into training as possible, and then generalize within the possible variations. This paper proposes a principled approach where a context module is coevolved with a skill module. The context module recognizes the variation and modulates the skill module so that the entire system performs well in unseen situations. The approach is evaluated in a challenging version of the Flappy Bird game where the effects of the actions vary over time. The Context+Skill approach leads to significantly more robust behavior in environments with previously unseen effects. Such a principled generalization ability is essential in deploying autonomous agents in real world tasks, and can serve as a foundation for continual learning as well.

show abstract

Towards continual reinforcement learning through evolutionary meta-learning

Cited by 6 publications

References 2 publications

Safe Reinforcement Learning through Meta-learned Instincts

Safe Reinforcement Learning through Meta-learned Instincts

Generalization of Agent Behavior through Explicit Representation of Context

Adapting to Unseen Environments through Explicit Representation of Context

Contact Info

Product

Resources

About