The Role of the Basal Ganglia in Exploration in a Neural Model Based on Reinforcement Learning

Abstract

www.iitm.ac.inWe present a computational model of basal ganglia as a key player in exploratory behavior. The model describes exploration of a virtual rat in a simulated water pool experiment. The virtual rat is trained using a reward-based or reinforcement learning paradigm which requires units with stochastic behavior for exploration of the system's state space. We model the Subthalamic Nucleus-Globus Pallidus externa (STN-GPe) segment of the basal ganglia as a pair of neuronal layers with oscillatory dynamics, exhibiting a variety of dynamic regimes such as chaos, traveling waves and clustering. Invoking the property of chaotic systems to explore state-space, we suggest that the complex exploratory dynamics of STN-GPe system in conjunction with dopamine-based reward signaling from the Substantia Nigra pars compacta (SNc) present the two key ingredients of a reinforcement learning system.

Protective Effect of Melatonin upon Neuropathology, Striatal Function, and Memory Ability after Intracerebral Hemorrhage in Rats

“…ICH most commonly occurs in the basal ganglia (striatum), and lesions there are reported to cause significant learning and memory deficits (Bhatia and Marsden, 1994;Hochstenbach et al, 1998;Su et al, 2007;Werring et al, 2004). Neuropsychological studies confirm the role of the striatum in cognition (Benke et al, 2003;El Massioui et al, 2007;Ragozzino, 2007;Sridharan et al, 2006), and ICH into the caudate putamen of rats produces a similar effect (Hartman et al, 2009).…”

What do the basal ganglia do? A modeling perspective

“…We had earlier suggested that the STN-GPe loop inside the BG circuit is rightly placed to support exploratory behavior (Sridharan et al 2006). In RL models, exploration is always modeled by stochastic components.…”

“…Complex activity of STN-GPe loop in normal BG, and its loss in Parkinsonian conditions has been attributed a deep functional significance, and is interpreted as a source of the stochastic exploration required by RL (Sridharan et al 2006). The model of Sridharan et al (2006) describes a simulated Morris water pool experiment, wherein a virtual rat explores for a hidden platform with the help of visible landmarks.…”

Reinforcement Learning Embedded in Brains and Robots

“…In any case, the different parallel loops need to communicate, possibly via the indirect pathway of the basal ganglia. Exploration -Exploitation Sridharan, Prashanth and Chakravarthy (2006) address the problem that a RL network has to produce randomness in some controllable fashion, in order to produce stochastic action choices. For this purpose they implement an oscillatory circuit via reciprocal excitatory-inhibitory connections.…”

“…1) represents a suitable oscillatory circuit in that the STN excites the GPe and in turn receives inhibition. Together with short-range lateral inhibition the model produces chaotic oscillatory activity that becomes more regular only with stronger input from the cortex (Sridharan et al,2006). They propose that, in case of weak or novel sensory input, the irregular firing causes an agent to behave more randomly and thereby to explore the state space.…”