Habits as action sequences: hierarchical action control and changes in outcome value

Abstract

One contribution of 18 to a Theme Issue 'The principles of goal-directed decisionmaking: from neural mechanisms to computation and robotics'. Goal-directed action involves making high-level choices that are implemented using previously acquired action sequences to attain desired goals. Such a hierarchical schema is necessary for goal-directed actions to be scalable to real-life situations, but results in decision-making that is less flexible than when action sequences are unfolded and the decision-maker deliberates step-by-step over the outcome of each individual action. In particular, from this perspective, the offline revaluation of any outcomes that fall within action sequence boundaries will be invisible to the high-level planner resulting in decisions that are insensitive to such changes. Here, within the context of a two-stage decisionmaking task, we demonstrate that this property can explain the emergence of habits. Next, we show how this hierarchical account explains the insensitivity of over-trained actions to changes in outcome value. Finally, we provide new data that show that, under extended extinction conditions, habitual behaviour can revert to goal-directed control, presumably as a consequence of decomposing action sequences into single actions. This hierarchical view suggests that the development of action sequences and the insensitivity of actions to changes in outcome value are essentially two sides of the same coin, explaining why these two aspects of automatic behaviour involve a shared neural structure.

Hippocampus and subregions of the dorsal striatum respond differently to a behavioral strategy change on a spatial navigation task

“…Automating behavior is a way to optimize benefit from an environment. Thus situations (e.g., stimuli) associated with actions that have consistently led to reinforcement eventually come to release appropriate action chains (Adams 1982;Daw et al 2006;Dezfouli et al 2014; These stimulus-action associations are cached and controlled by sensorimotor circuits in the basal ganglia, such as the aDLS (Everitt and Robbins 2013;Graybiel 1998Graybiel , 2008Miyachi et al 2002;Packard and McGaugh 1996;Yin and Knowlton 2006).…”

“…In contrast, in stable environments, as actions become more automated (e.g., habit based), control shifts to sensorimotor circuits capable of encoding action chains (Dezfouli et al 2014;Graybiel 2008;Yin and Knowlton 2006), which include the aDLS (Alexander and Crutcher 1990;Berendse et al 1992;Carrelli and West 1991;McGeorge and Faull 1989). Graybiel and colleagues have reported that the development of these action chains on a cued T-maze aligns with the development of preferential firing in dorsolateral striatum at the beginning and end of their T-maze (task bracketing; Barnes et al 2005;Jog et al 1999;Smith and Graybiel 2013;Thorn et al 2010).…”

The algorithmic anatomy of model-based evaluation

“…Thus, the MF prediction arises from learning from past encounters of state x to minimize untoward inconsistencies. Even more extreme MF systems directly couple observations to decisions, for instance using the inconsistencies to learn appropriate actions or action sequences [9][10][11].…”

Consumers and their behavior: state of the art in behavioral science supporting use phase modeling in LCA and ecodesign

“…Habits are learned behavioral patterns that are performed without much conscious deliberation and instead strongly controlled by the stimulus environment (Dezfouli et al 2014;Orbell and Verplanken 2015). Indeed, actions such as turning the light on/off, disposing waste, running the bath or shower are often done in a repetitive and almost unconscious way.…”