Isotropic sequence order learning using a novel linear algorithm in a closed loop behavioural system

Abstract

In this article, we present an isotropic algorithm for sequence order learning. Its central goal is to learn the causal relation between two (or more) inputs in order to react to the earliest incoming signal after successful learning (like in typical classical conditioning situations). We implement this algorithm in a behaving system (a robot) thereby creating a closed loop situation where the learner's actions influence its own sensor inputs to the end of creating an autonomous agent. Autonomous behaviour implies that learning goals are internally defined within the organism's capabilities. Standard learning models for sequence learning (e.g., TDlearning) need an externally defined reward. This, however, is in conflict with the requirement of an implicitly defined internal goal in autonomous behaviour. Therefore, in this study we present a system in which the external reward is replaced by a reflex loop. This loop explicitly includes the environment. Every reflex loop has the inherent disadvantage which is that its re-actions occur each time just after a reflex-eliciting sensor event and thus 'too late'. However, a reflex can serve as the internal reference for sequence order learning which has the task of eliminating this disadvantage by creating earlier anticipatory actions. In our system learning is achieved by modifying synaptic weights of a linear neuron with a correlation based learning rule which involves the derivative of the neuron's output. All input lines are entirely isotropic. The synaptic weight change curve of this rule is strongly related to the temporal Hebb learning rule which was found in spike timing experiments. We find that after learning the reflex loop is replaced in functional terms with an earlier anticipatory action (and pathway). In addition, we observed that the synaptic weights stabilise as soon as the reflex remains silent.

Temporal Sequence Learning, Prediction, and Control: A Review of Different Models and Their Relation to Biological Mechanisms

“…Rate-based models rely on the average firing rate of a neuron and not on its precise firing time (Sutton and Barto, 1981;Kosco, 1986;Klopf, 1986Klopf, , 1988Kempter et al, 2001b;Porr and Wörgötter, 2002;Kistler, 2002;Porr and Wörgötter, 2003a). As such they are at first sight unrelated to the spike-based models.…”

Sign and Relevance learning

“…The learning terminates when A P fends off the disturbance at node 3 precisely and persistently. In this case, the reflex loop is no longer evoked, E remains at zero, the learner undergoes no further changes, and thus, the SaR network has successfully generated the forward model of the reflex, in a similar fashion to the model-based learning paradigms presented in the work by Porr and Wörgötter (2002, 2003.…”

“…This lead to the assumption that the brain resembles an actor / critic architecture where dopamine, as the reward prediction error, drives synaptic changes in the striatum (Humphries et al, 2006). An-other interpretation of the actor / critic architecture is a nested closed-loop platform where an inner reflex loop generates an error signal which tunes an actor in an outer loop to create anticipatory actions, in other words, the actor generates a forward model of the reflex (Porr and Wörgötter, 2002). Thus the actor / critic architecture can be used for both model and model free learning.…”

Isotropic-sequence-order learning in a closed-loop behavioural system

“…Characteristic waveforms of the system in response to δ-pulse inputs are shown in figure 2a. The weight change can be calculated analytically by correlating the two resonator responses of H 0 and H 1 in the Laplace space (Porr & Wörgötter 2002). Figure 2b shows how the synaptic weight ρ 1 of the conditioned input changes, assuming identical bandpass filters for two inputs which occur with a time difference of T between them.…”