Temporal Sequence Learning, Prediction, and Control: A Review of Different Models and Their Relation to Biological Mechanisms

Abstract

In this article we compare methods for temporal sequence learning (TSL) across the disciplines machine-control, classical conditioning, neuronal models for TSL as well as spiketiming dependent plasticity. This review will briefly introduce the most influential models and focus on two questions: 1) To what degree are reward-based (e.g. TD-learning) and correlation based (hebbian) learning related? and 2) How do the different models correspond to possibly underlying biological mechanisms of synaptic plasticity? We will first compare the different models in an open-loop condition, where behavioral feedback does not alter the learning. Here we observe, that reward-based and correlation based learning are indeed very similar. Machine-control is then used to introduce the problem of closed-loop control (e.g. "actor-critic architectures"). Here the problem of evaluative ("rewards") versus nonevaluative ("correlations") feedback from the environment will be discussed showing that both learning approaches are fundamentally different in the closed-loop condition. In trying to answer the second question we will compare neuronal versions of the different learning architectures to the anatomy of the involved brain structures (basal-ganglia, thalamus and cortex) and to the molecular biophysics of glutamatergic and dopaminergic synapses. Finally we discuss the different algorithms used to model spike-timing dependent plasticity (STDP) and compare them to reward based learning rules. Certain similarities are found in spite of the strongly different time scales. Here we focus on the biophysics of the different Calcium-release mechanisms known to be involved in STDP.

Action and behavior: a free-energy formulation

“…This is closely related to perspectives on motor control and sequence learning that "minimize deviations from the desired state, that is, to minimize disturbances of the homeostasis of the feedback loop." See Wörgötter and Porr (2005) for a fuller discussion. In summary, avoiding surprise is fundamental for survival and speaks to the basic need of organisms to maintain equilibrium within their environment.…”

“…Promising about such approaches is that they naturally extend to distributed state representations and efficiently cope with uncertainty"; see Toussaint (2009) for a fuller discussion of probabilistic inference as a model of planned behavior. There are also alternative self-referenced schemes (e.g., Verschure and Voegtlin 1998;Wörgötter and Porr 2005;Tschacher and Haken 2007) that may have greater ethological and neuronal plausibility. This theme will be developed further in a forthcoming article on value-learning and freeenergy.…”

Computational models of reinforcement learning: the role of dopamine as a reward signal

“…(4) A model of the environment which includes a representation of the environment dynamics required for maximizing the sum of future rewards. These general RL concepts are further explained and elaborated by various authors (Dayan and Daw 2008;Kaelbling et al 1996;Montague et al 2004a;Sutton and Barto 1998;Worgotter and Porr 2005;Dayan and Abbott 2001).…”

Back to optimality: a formal framework to express the dynamics of learning optimal behavior

“…Given its generality (Wo¨rgo¨tter & Porr, 2005), we take reinforcement learning as our basic framework and express it in terms of optimal control systems. At the end of the day, animals are behavior systems-sets of behaviors that are organized around biological functions and goals, e.g., feeding (Timberlake & Silva, 1995), defense (Fanselow, 1994), or sex (Domjan, 1994).…”