The Neuromodulatory System: A Framework for Survival and Adaptive Behavior in a Challenging World

Abstract

Biological organisms have the ability to respond quickly to an ever-changing world. Because this adaptability is so critical for survival, all vertebrates have sub-cortical structures, which comprise the neuromodulatory systems, to regulate fundamental behavior and drive decision making in response to environmental events. In the vertebrate, there are separate neuromodulators that respond to threats, reward anticipation, novelty, and attentional effort. However, each of these neuromodulatory systems has a similar effect, that is, to cause an organism to be decisive when environmental conditions call for such actions, and allow the organism to be more exploratory when there are no pressing events. In this article, it is proposed that principles of the neuromodulatory system could provide a framework for controlling artificial agents that may improve current artificial agent behavior. These agents would operate autonomously, effectively explore their environment, and be decisive when environmental conditions call for action.

Evolution of recollection and prediction in neural networks

“…We expect results comparable to those reported here, and also to those in [21]. Furthermore, actually modeling how the external memory became internalized would be an intriguing topic (a hint from the neuromodulation research such as [34] could provide the necessary insights). Insights gained from evolving an arbitrary neural network topology may also be helpful [39], [40].…”

“…Furthermore, neurogenesis is most often observed in the hippocampus and in the olfactory bulb, alluding to a close functional demand [33]. Finally, it is interesting to think of neuromodulators [34] as a form of internal marker dropping, in the fashion explored in this paper.…”

From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation

“…Positive modulation increases the difference between the two weights and negative modulation decreases the difference, thereby decreasing the overall strength of the pathway. A low modulatory value, or tonic value, implies small changes and is useful in conditions of low relevance for learning, as opposed to phasic values (Aston-Jones and Cohen, 2005;Krichmar, 2008). A crucial aspect in the present implementation is that under negative modulation the pathway oscillates between weakly excitatory and inhibitory states due to the stabilizing effect of anti-Hebbian plasticity.…”

“…The alternation of these two regimes of Hebbian and anti-Hebbian plasticity produces the key dynamics of alternating exploitation and exploration observed in operant reward learning. The change in modulatory activity has in fact been suggested to regulate the alternation of exploration and exploitation in Krichmar (2008). Thus, while the dynamics of modulated Hebbian plasticity and modulated spike-timedependent plasticity (STDP) have been extensively investigated (Abbott, 1990;Montague et al, 1996;Florian, 2007;Porr and Wörgötter, 2007;Frémaux et al, 2010;Pfeiffer et al, 2010), the novelty of this work is their extension by means of saturation and noise, resulting in a simpler and more fundamental connection between local changes and higher-level simulated behavior.…”

“…Moreover, modulation appears to regulate a large variety of behaviors like arousal, attention, reward learning, and memory (HarrisWarrick and Marder, 1991;Hasselmo, 1995;Aston-Jones and Cohen, 2005), resulting in an accordingly large spectrum of dynamics and models that regulate synaptic efficacy, synaptic changes and other neural variables (Hasselmo and Schnell, 1994;Fellous and Linster, 1998;Doya, 2002;Smith et al, 2002;Krichmar, 2008;Cox and Krichmar, 2009). One promising computational aspect of modulation is the possibility of increasing, decreasing or inverting the strength and sign of plasticity (Abbott, 1990;Montague et al, 1996;Florian, 2007;Porr and Wörgötter, 2007;Izhikevich, 2007;Pfeiffer et al, 2010), making neuromodulation particularly suitable for modeling and implementing learning processes (Sporns and Alexander, 2002;Doya, 2002;Doya and Uchibe, 2005;Soula et al, 2005;Farries and Fairhall, 2007;Krichmar, 2008;Cox and Krichmar, 2009). The focus in this study is on this latter role of modulation as a gating mechanism for Hebbian synaptic plasticity.…”