Optimal habits can develop spontaneously through sensitivity to local cost

Abstract

Habits and rituals are expressed universally across animal species. These behaviors are advantageous in allowing sequential behaviors to be performed without cognitive overload, and appear to rely on neural circuits that are relatively benign but vulnerable to takeover by extreme contexts, neuropsychiatric sequelae, and processes leading to addiction. Reinforcement learning (RL) is thought to underlie the formation of optimal habits. However, this theoretic formulation has principally been tested experimentally in simple stimulus-response tasks with relatively few available responses. We asked whether RL could also account for the emergence of habitual action sequences in realistically complex situations in which no repetitive stimulus-response links were present and in which many response options were present. We exposed naïve macaque monkeys to such experimental conditions by introducing a unique free saccade scan task. Despite the highly uncertain conditions and no instruction, the monkeys developed a succession of stereotypical, self-chosen saccade sequence patterns. Remarkably, these continued to morph for months, long after session-averaged reward and cost (eye movement distance) reached asymptote. Prima facie, these continued behavioral changes appeared to challenge RL. However, trial-by-trial analysis showed that pattern changes on adjacent trials were predicted by lowered cost, and RL simulations that reduced the cost reproduced the monkeys' behavior. Ultimately, the patterns settled into stereotypical saccade sequences that minimized the cost of obtaining the reward on average. These findings suggest that brain mechanisms underlying the emergence of habits, and perhaps unwanted repetitive behaviors in clinical disorders, could follow RL algorithms capturing extremely local explore/exploit tradeoffs.free-viewing | naïve monkey | reinforcement learning | saccade R einforcement learning (RL) theory formalizes the process by which rewards and punishments can shape the behaviors of a goal-seeking agent-person, animal, or robot-toward optimality (1). RL algorithms have been widely applied in neuroscience to characterize neural activity in animals and human subjects, most famously for the dopamine-containing systems of the brain and related brain regions (2-5). These ideas have also been influential in the study of habit learning, in which habits are typically thought to arise when behaviors, through repetition, eventually become reinforcement-independent, stimulus-response (S-R) associations that can be executed in a semiautomatic manner (6).In most learning experiments designed to test these ideas, a small range of relationships between actions and reward is imposed, cost-benefit ratios are explicit, and fixed and usually limited numbers of response choices are available, as for example when human subjects are asked to move a cursor in one direction to receive a monetary reward in a computer game, or when rodents are trained to press one or a small set of levers to receive a food reward. RL algorithms of va...

Neuronal activity in the primate dorsomedial prefrontal cortex contributes to strategic selection of response tactics

“…The monkeys could then have taken advantage of this stimulusresponse relationship and developed a habit of automatically determining the reaching direction from the location of the cues. Such a change in the monkeys' tactics conforms to switching from controlled to automatic action (19,31). This interpretation is supported by the decreased reaction time under the single-tactic condition (Fig.…”

“…When performing highly practiced or routine motor behaviors, we often attempt to enhance efficiency by using a variety of tactics to select an action (19). The selection of an action and the selection of tactics differ in that the former is the process of determining what to do, whereas the latter reflects the internal protocol of how to decide what to do.…”

Humans have idiosyncratic and task-specific scanpaths for judging faces

“…For both monkeys, this initial scanpath routine gradually became more efficient, but only one of the monkeys developed the optimal scanpath routine. Both Rehder and Hoffman (2005) and Desrochers et al (2010) have been successfully modeled using reinforcement learning algorithms (Desrochers et al, 2010;Nelson & Cottrell, 2007). Similar mechanisms are thought to be implemented in the basal ganglia, which enables scanpath routines to be learned in humans and other animals (Hayhoe & Ballard, 2005;Hikosaka, Takikawa, & Kawagoe, 2000).…”

Learning optimal strategies in complex environments

“…Dopamine neurons in the brainstem predict future rewards consistent with temporaldifference reinforcement learning (6,7). Neurons in the cortex reflect these reward predictions and are sensitive to trial-bytrial fluctuations (8), which could drive the exploration of different regular scan paths (2). Other domains where reinforcement learning has been found to be effective include birdsong learning (9) and finding the optimal Nash equilibrium in games played against an opponent (10).…”

“…In the study by Desrochers et al (2), a monkey was free to scan an array of dots, one of which was randomly baited with a reward on each trial. After several sessions of learning, and without any instructions, the monkey quickly settled on a regular scan path that visited all of the dots once on each scan out of the infinite number of possible scan paths that the monkey could have adopted.…”