Deep Reinforcement Learning With Modulated Hebbian Plus Q-Network Architecture

Abstract

In this paper, we consider a subclass of partially observable Markov decision process (POMDP) problems which we termed confounding POMDPs. In these types of POMDPs temporal difference (TD)-based RL algorithms struggle, as TD error cannot be easily derived from observations. We solve these types of problems using a new bio-inspired neural architecture that combines a modulated Hebbian network (MOHN) with DQN, which we call modulated Hebbian plus Q network architecture (MOHQA). The key idea is to use a Hebbian network with rarely correlated bio-inspired neural traces to bridge temporal delays between actions and rewards when confounding observations and sparse rewards result in inaccurate TD errors. In MOHQA, DQN learns low-level features and control, while the MOHN contributes to the high-level decisions by associating rewards with past states and actions. Thus the proposed architecture combines two modules with significantly different learning algorithms, a Hebbian associative network and a classical DQN pipeline, exploiting the advantages of both. Simulations on a set of POMDPs and on the Malmo environment show that the proposed algorithm improved DQN's results and even outperformed control tests with advantage-actor critic (A2C), Quantile regression DQN with long short term memory (QRDQN+LSTM), Monte-Carlo policy gradient (REINFORCE) and aggregated memory for reinforcement learning (AMRL) algorithms on most difficult POMDPs with confounding stimuli and sparse rewards.

Context Meta-Reinforcement Learning via Neuromodulation

“…The continuous control environments are the simple 2D navigation, the half-cheetah direction (Finn et al, 2017) and velocity (Finn et al, 2017) Mujoco (Todorov et al, 2012) based environments and the meta-world ML1 and ML45 environments (Yu et al, 2020). The discrete action environment is a graph navigation environment that supports configurable levels of complexity called the CTgraph (Soltoggio et al, 2021;Ladosz et al, 2021;Ben-Iwhiwhu et al, 2020). The experimental setup focused on investigating the beneficial effect of the proposed neuromodulatory mechanism when augmenting existing meta-RL frameworks (i.e., neuromodulation as complementary tool to meta-RL rather than competing).…”

A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems

“…Importantly, STELLAR integrated 11 innovative components that solve different challenges and requirements for LL. It employed Sliced Cramer Preservation (SCP) (Kolouri et al, 2020), or the sketched version of it (SCP++) (Li et al, 2021), and Complex Synapse Optimizer (Benna and Fusi, 2016) to overcome catastrophic forgetting of old tasks; Self-Preserving World Model (Ketz et al, 2019) and Context-Skill Model (Tutum et al, 2021) for backward transfer to old tasks as well as forward transfer to their variants; Neuromodulated Attention (Zou et al, 2020) for rapid performance recovery when an old task repeats; Modulated Hebbian Network (Ladosz et al, 2022) and Plastic Neuromodulated Network (Ben-Iwhiwhu et al, 2021) for rapid adaptation to new tasks; Reflexive Adaptation (Maguire et al, 2021) and Meta-Learned Instinct Network (Grbic and Risi, 2021) to safely adapt to new tasks; and Probabilistic Program Neurogenesis (Martin and Pilly, 2019) to scale up the learning of new tasks during fielded operation. More details on the precise effect of each of these components are beyond the scope of this paper; however, this case study outlines how the integrated system dynamics demonstrated LL using the proposed metrics, and how these metrics shaped the advancement of the SG-HRL system.…”

Application of Machine Learning Technologies for Transport layer Congestion Control

“…[19]. An RL-based architecture proposed by the researchers in [20] may be used to decouple model development and implementation. The CC framework is proposed offline and deployed in authentic for online display alteration decision-making in instantaneously.…”

“…Model of Deep Reinforcement Learning (DRL) for congestion control MPTCP's route diversity-induced decrease of Controllability is the topic of [19]. An RL-based architecture proposed by the researchers in[20] may be used to decouple model development and implementation. The CC framework is proposed offline and deployed in authentic for online display alteration decision-making in instantaneously.…”