Modelling Coordination of Learning Systems: A Reservoir Systems Approach to Dopamine Modulated Pavlovian Conditioning

Lowe, Robert; Mannella, Francesco; Ziemke, Tom; Baldassarre, Gianluca

doi:10.1007/978-3-642-21283-3_51

Cited by 4 publications

(15 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the one hand, the "wiring" is not driven by the firing activities of the neurons but by their rates of change. This is reminiscent of differential Hebbian learning studied in earlier work, see Kosko (1986), Klopf (1988), Roberts (1999), and Lowe et al (2011). The advantage of differential over pure Hebbian learning for the self-organized behavior acquisition has been discussed in a concrete setting close to that of this paper in Der and Martius (2015).…”

Section: Introductionmentioning

confidence: 77%

“…When learning this controller with a Hebbian law, the rate of change 1 of synapse C ij would be proportional to the input x j into the synapse j of neuron i multiplied by its activation y i , i.e.,Ċ ij ∝ y i x j . Differential Hebbian learning on its hand would use the rates of change, i.e.,Ċ ij ∝ẏ iẋj , see Kosko (1986), Klopf (1988), Roberts (1999), and Lowe et al (2011). However, this must be modified in order to establish the contact with the external world.…”

Section: Synaptic Plasticitymentioning

confidence: 99%

See 1 more Smart Citation

In Search for the Neural Mechanisms of Individual Development: Behavior-Driven Differential Hebbian Learning

Der

2016

Front. Robot. AI

View full text Add to dashboard Cite

When Donald Hebb published his 1949 book "The Organization of Behavior" he opened a new way of thinking in theoretical neuroscience that, in retrospective, is very close to contemporary ideas in self-organization. His metaphor of "wiring" together what "fires together" matches very closely the common paradigm that global organization can derive from simple local rules. While ingenious at his time and inspiring the research over decades, the results still fall short of the expectations. For instance, unsupervised as they are, such neural mechanisms should be able to explain and realize the self-organized acquisition of sensorimotor competencies. This paper proposes a new synaptic law that replaces Hebb's original metaphor by that of "chaining together" what "changes together." Starting from differential Hebbian learning, the new rule grounds the behavior of the agent directly in the internal synaptic dynamics. Therefore, one may call this a behavior-driven synaptic plasticity. Neurorobotics is an ideal testing ground for this new, unsupervised learning rule. This paper focuses on the close coupling between body, control, and environment in challenging physical settings. The examples demonstrate how the new synaptic mechanism induces a self-determined "search and converge" strategy in behavior space, generating spontaneously a variety of sensorimotor competencies. The emerging behavior patterns are qualified by involving body and environment in an irreducible conjunction with the internal mechanism. The results may not only be of immediate interest for the further development of embodied intelligence. They also offer a new view on the role of self-learning processes in natural evolution and in the brain. Videos and further details may be found under http://robot.informatik.uni-leipzig.de/research/ supplementary/NeuroAutonomy/.

show abstract

Section: Introductionmentioning

confidence: 77%

Section: Synaptic Plasticitymentioning

confidence: 99%

In Search for the Neural Mechanisms of Individual Development: Behavior-Driven Differential Hebbian Learning

Der

2016

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…The work of Alexander and Sporns (2002) demonstrates the effects of sensorimotor feedback on a rewardbased network. By using a reward prediction learning algorithm producing data analogous to the diffusive DA projections from the VTA affecting PFC and motor cortex (similar to Lowe et al 2009b described in the previous subsection), they demonstrated that robots were able to utilise this system to produce unforeseen environment-dependent clustering foraging strategies. This example of a 'closed loop' paradigm provides an example of how organisms in the real world may modulate their own behaviour using reward-based systems -the robots can affect the timing of the onset of the rewarding stimuli and reward-predictive stimuli (see Alexander 2002, 2003 for other such experimental work using disembodied and embodied reward-based networks).…”

Section: Cognitive Robotics: Investigations Of Value Discrimination Amentioning

confidence: 92%

“…One aspect of the model of Lowe et al (2009b) that is lacking is a component that accounts for processing of spatial information in the environment, which is naturally of paramount importance in scenarios where the environment can be seen as part of the cognitive system (cf. Wilson 2002).…”

Section: Candidate Neurocomputational Models For Core Neurophysiologimentioning

confidence: 99%

“…The CS and US bars at the top illustrate onset or omission of expected rewards, the activity below illustrates activity in the VTA and AMG according to positive (burst) or negative (dip) reward prediction errors. In this particular simulation second-order conditioning is occurring across trials 21, 41 and 599, respectively (see Lowe et al 2009b for details). of its CS-US onset profile, i.e.…”

Section: Candidate Neurocomputational Models For Core Neurophysiologimentioning

confidence: 99%

See 1 more Smart Citation

Towards a cognitive robotics methodology for reward-based decision-making: dynamical systems modelling of the Iowa Gambling Task

Lowe

Ziemke

2010

Connection Science

Self Cite

View full text Add to dashboard Cite

The somatic marker hypothesis (SMH) posits that the role of emotions and mental states in decision-making manifests through bodily responses to stimuli of import to the organism's welfare. The Iowa Gambling Task (IGT), proposed by Bechara and Damasio in the mid-1990s, has provided the major source of empirical validation to the role of somatic markers in the service of flexible and cost-effective decision-making in humans. In recent years the IGT has been the subject of much criticism concerning: (1) whether measures of somatic markers reveal that they are important for decision-making as opposed to behaviour preparation;(2) the underlying neural substrate posited as critical to decision-making of the type relevant to the task; and (3) aspects of the methodological approach used, particularly on the canonical version of the task. In this paper, a cognitive robotics methodology is proposed to explore a dynamical systems approach as it applies to the neural computation of reward-based learning and issues concerning embodiment. This approach is particularly relevant in light of a strongly emerging alternative hypothesis to the SMH, the reversal learning hypothesis, which links, behaviourally and neurocomputationally, a number of more or less complex reward-based decision-making tasks, including the 'A-not-B' task -already subject to dynamical systems investigations with a focus on neural activation dynamics. It is also suggested that the cognitive robotics methodology may be used to extend systematically the IGT benchmark to more naturalised, but nevertheless controlled, settings that might better explore the extent to which the SMH, and somatic states per se, impact on complex decision-making.

show abstract

Novel plasticity rule can explain the development of sensorimotor intelligence

Der¹,

Martius²

2015

Proc. Natl. Acad. Sci. U.S.A.

View full text Add to dashboard Cite

Grounding autonomous behavior in the nervous system is a fundamental challenge for neuroscience. In particular, self-organized behavioral development provides more questions than answers. Are there special functional units for curiosity, motivation, and creativity? This paper argues that these features can be grounded in synaptic plasticity itself, without requiring any higher-level constructs. We propose differential extrinsic plasticity (DEP) as a new synaptic rule for self-learning systems and apply it to a number of complex robotic systems as a test case. Without specifying any purpose or goal, seemingly purposeful and adaptive rhythmic behavior is developed, displaying a certain level of sensorimotor intelligence. These surprising results require no systemspecific modifications of the DEP rule. They rather arise from the underlying mechanism of spontaneous symmetry breaking, which is due to the tight brain body environment coupling. The new synaptic rule is biologically plausible and would be an interesting target for neurobiological investigation. We also argue that this neuronal mechanism may have been a catalyst in natural evolution.R esearch in neuroscience produces an understanding of the brain on many different levels. At the smallest scale, there is enormous progress in understanding mechanisms of neural signal transmission and processing (1-4). At the other end, neuroimaging and related techniques enable the creation of a global understanding of the brain's functional organization (5, 6). However, a gap remains in binding these results together, which leaves open the question of how all these complex mechanisms interact (7-9). This paper advocates for the role of self-organization in bridging this gap. We focus on the functionality of neural circuits acquired during individual development by processes of self-organizationmaking complex global behavior emerge from simple local rules. Donald Hebb's formula "cells that fire together wire together" (10) may be seen as an early example of such a simple local rule which has proven successful in building associative memories and perceptual functions (11,12). However, Hebb's law and its successors like BCM (13) and STDP (14, 15) are restricted to scenarios where the learning is driven passively by an externally generated data stream. However, from the perspective of an autonomous agent, sensory input is mainly determined by its own actions. The challenge of behavioral self-organization requires a new kind of learning that bootstraps novel behavior out of the self-generated past experiences. This paper introduces a rule which may be expressed as "chaining together what changes together." This rule takes into account temporal structure and establishes contact to the external world by directly relating the behavioral level to the synaptic dynamics. These features together provide a mechanism for bootstrapping behavioral patterns from scratch.This synaptic mechanism is neurobiologically plausible and raises the question of whether it is present in living beings...

show abstract

Modelling Coordination of Learning Systems: A Reservoir Systems Approach to Dopamine Modulated Pavlovian Conditioning

Cited by 4 publications

References 10 publications

In Search for the Neural Mechanisms of Individual Development: Behavior-Driven Differential Hebbian Learning

In Search for the Neural Mechanisms of Individual Development: Behavior-Driven Differential Hebbian Learning

Towards a cognitive robotics methodology for reward-based decision-making: dynamical systems modelling of the Iowa Gambling Task

Novel plasticity rule can explain the development of sensorimotor intelligence

Contact Info

Product

Resources

About