From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation

Abstract

Citation: SOLTOGGIO, A. and STANLEY, K.O., 2012. From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation. Neural Networks, 34 pp. 28-41.Additional Information:• NOTICE: this is the author's version of a work that was accepted for publication in Neural Networks. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. A definitive version was subsequently published in Neural Networks, vol. 34 (2012) AbstractSynaptic plasticity is a major mechanism for adaptation, learning and memory. Yet current models struggle to link local synaptic changes to the acquisition of behaviors. The aim of this paper is to demonstrate a computational relationship between local Hebbian plasticity and behavior learning by exploiting two traditionally unwanted features: neural noise and synaptic weight saturation. A modulation signal is employed to arbitrate the sign of plasticity: when the modulation is positive, the synaptic weights saturate to express exploitative behavior; when it is negative, the weights converge to average values and neural noise reconfigures the network's functionality. This process is demonstrated through simulating neural dynamics in the autonomous emergence of fearful and aggressive navigating behaviors and in the solution to reward-based problems. The neural model learns, memorizes and modifies different behaviors that lead to positive modulation in a variety of settings. The algorithm establishes a simple relationship between local plasticity and behavior learning by demonstrating the utility of noise and weight saturation. Moreover it provides a new tool to simulate adaptive behavior and contributes to bridging the gap between synaptic changes and behavior in neural computation.

Solving the Distal Reward Problem with Rare Correlations

“…For example, the modulator acetylcholine was shown to enhance and stabilize learning and memory (for a review of related studies, see Bailey et al, 2000), revealing the central role of modulation in regulating long-and shortterm plasticity (Kandel & Tauc, 1965). These findings provided inspiration for neural models that use neuromodulation to gate neural plasticity such that high modulation reinforces actions, while low modulation extinguishes actions (Abbott, 1990;Montague, Dayan, Person, & Sejnowski, 1995;Fellous & Linster, 1998;Porr & Wörgötter, 2007;Alexander & Sporns, 2002;Soula, Alwan, & Beslon, 2005;Florian, 2007;Pfeiffer, Nessler, Douglas, & Maass, 2010;Soltoggio & Stanley, 2012). Modulatory neurons were also shown to appear spontaneously in the evolution of artificial neural networks as evolutionarily advantageous traits in learning and memory tasks (Soltoggio, Bullinaria, Mattiussi, Dürr, & Floreano, 2008).…”

Real-time Hebbian Learning from Autoencoder Features for Control Tasks

“…Neuromodulation enables the increase, decrease, or reversal of Hebbian plasticity according to feedback from the environment. Models augmented with neuromodulation have been shown to implement a variety of typical features of animal operant learning such as reinforcement of rewarding actions, extinction of unproductive actions, and behavior reversal (Soltoggio and Stanley, 2012;. The combination of Hebbian ANNs with neuromodulatory signals in recent years has especially inspired neuroevolution and artificial life researchers by opening up the possibility of evolving ANNs that can learn from a sequence of rewards over their lifetime (Soltoggio et al, 2008(Soltoggio et al, , 2007Soltoggio and Jones, 2009;Soltoggio and Stanley, 2012;Risi and Stanley, 2012;Coleman and Blair, 2012).…”

“…As a medium for adaptation and learning, neural plasticity has long captivated artificial life and related fields (Baxter, 1992;Floreano and Urzelai, 2000;Niv et al, 2002;Soltoggio et al, 2008Soltoggio et al, , 2007Soltoggio and Jones, 2009;Soltoggio and Stanley, 2012;Risi and Stanley, 2010;Risi et al, 2011;Risi and Stanley, 2012;Stanley et al, 2003;Coleman and Blair, 2012). Much of this body of research focuses on Hebbian-inspired rules that change the weights of connections in proportion to the correlation of source and target neuron activations (Hebb, 1949).…”

“…The simplicity of such Hebbian-inspired rules makes them easy and straightforward to integrate into larger systems and experiments, such as investigations into the evolution of plastic neural networks (Floreano and Urzelai, 2000;Soltoggio et al, 2008;Risi et al, 2011). Thus they have enabled inquiry into such diverse problems as task switching (Floreano and Urzelai, 2000), neuromodulation (Soltoggio et al, 2008), the evolution of memory (Risi et al, 2011), and reward-mediated learning (Soltoggio and Stanley, 2012).…”

Learning the rules of a game: Neural conditioning in human-robot interaction with delayed rewards

“…neuromodulation [22], it drives plasticity to reinforce rewardrelated stimuli and actions. The RCHP rule is given as…”