Evolving inborn knowledge for fast adaptation in dynamic POMDP problems

Abstract

Rapid online adaptation to changing tasks is an important problem in machine learning and, recently, a focus of meta-reinforcement learning. However, reinforcement learning (RL) algorithms struggle in POMDP environments because the state of the system, essential in a RL framework, is not always visible. Additionally, handdesigned meta-RL architectures may not include suitable computational structures for specific learning problems. The evolution of online learning mechanisms, on the contrary, has the ability to incorporate learning strategies into an agent that can (i) evolve memory when required and (ii) optimize adaptation speed to specific online learning problems. In this paper, we exploit the highly adaptive nature of neuromodulated neural networks to evolve a controller that uses the latent space of an autoencoder in a POMDP. The analysis of the evolved networks reveals the ability of the proposed algorithm to acquire inborn knowledge in a variety of aspects such as the detection of cues that reveal implicit rewards, and the ability to evolve location neurons that help with navigation. The integration of inborn knowledge and online plasticity enabled fast adaptation and better performance in comparison to some non-evolutionary meta-reinforcement learning algorithms. The algorithm proved also to succeed in the 3D gaming environment Malmo Minecraft.

Evolving and merging hebbian learning rules

“…Extending the evolved learning rules to be able to take into account reward signals could greatly improve the model's ability to respond to changes in the environment in an adaptive manner, and it is something we look forward to implementing in future studies. Something similar to this has been explored in other approaches to plastic networks [2,4,13,53], and we expect this to also be a beneficial addition to our approach.…”

Context Meta-Reinforcement Learning via Neuromodulation

“…The continuous control environments are the simple 2D navigation, the half-cheetah direction (Finn et al, 2017) and velocity (Finn et al, 2017) Mujoco (Todorov et al, 2012) based environments and the meta-world ML1 and ML45 environments (Yu et al, 2020). The discrete action environment is a graph navigation environment that supports configurable levels of complexity called the CTgraph (Soltoggio et al, 2021;Ladosz et al, 2021;Ben-Iwhiwhu et al, 2020). The experimental setup focused on investigating the beneficial effect of the proposed neuromodulatory mechanism when augmenting existing meta-RL frameworks (i.e., neuromodulation as complementary tool to meta-RL rather than competing).…”

Introducing lifelong machine learning in the active tuberculosis classification through chest radiographs

“…• Network growth strategies: based on adding parameters to the network in order to allow the learning of new tasks and, at the same time, retain previous knowledge through the old parameters [34], [35], [36]. • Selective plasticity and neuromodulation strategies: methods that allow a subgroup of parameters to be adjusted to the new task while another subgroup is almost or completely frozen [37], [38], [39], [40] . • Generative strategies: inspired by the existence of a dual memory system based on the hippocampus and neocortex in the human brain.…”