Balancing control: A Bayesian interpretation of habitual and goal-directed behavior

Schwöbel, Sarah; Marković, Dimitrije; Smolka, Michael N.; Kiebel, Stefan J.

doi:10.1016/j.jmp.2020.102472

Cited by 19 publications

(40 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using computational simulations, Miller and colleagues (2019) recently proposed that a frequency-based habitization process could reproduce hallmark findings of choice and perseveration in the experimental literature on habits (Guthrie, 1959; see also Schwöbel et al, 2021, for a Bayesian implementation of repetition-based habit learning). This view converges with evidence for frequency-related processes playing a role in various psychological processes.…”

Section: Characterizing Human Habits In the Labmentioning

confidence: 99%

Characterizing human habits in the lab

Nebe¹,

Kretzschmar²,

Tobler³

2022

Preprint

View full text Add to dashboard Cite

Habits pose a fundamental puzzle for those aiming to understand human behavior. They pervade our everyday lives and dominate some forms of psychopathology but are extremely hard to elicit in the lab. In this Registered Report, we develop novel experimental paradigms grounded in computational models, which suggest that habit strength should be proportional to the frequency of behavior and, in contrast to previous research, independent of value. Specifically, we manipulate how often participants perform responses in two tasks varying action repetition without, or separately from, variations in value. Moreover, we ask how this frequency-based habitization relates to value-based operationalizations of habit and self-reported propensities for habitual behavior in real life. (This is the first stage of a Registered Report in-principle-accepted by Collabra: Psychology.)

show abstract

Section: Characterizing Human Habits In the Labmentioning

confidence: 99%

Characterizing human habits in the lab

Nebe¹,

Kretzschmar²,

Tobler³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While the hierarchical model proposed by Dezfouli and Balleine has already been shown to reproduce experimental data of the two-stage task (Dezfouli and Balleine, 2013), it will be interesting to assess whether it can replicate key aspects of the present study. In a third approach, Schwöbel et al (2021) proposed a hierarchical Bayesian model that combines the idea of habit acquisition through repetition and habits as action sequences. In this model, habits are considered precise priors over action sequences in a Bayesian integrator model, where the value-based goal-directed mode of behavior is represented by a Bayesian likelihood function.…”

Section: Discussionmentioning

confidence: 99%

Interaction between habits as action sequences and goal-directed behavior under time pressure

Frölich

Esmeyer²,

Endraß³

et al. 2023

Front. Neurosci.

Self Cite

View full text Add to dashboard Cite

Human behavior consists in large parts of action sequences that are often repeated in mostly the same way. Through extensive repetition, sequential responses become automatic or habitual, but our environment often confronts us with events to which we have to react flexibly and in a goal-directed manner. To assess how implicitly learned action sequences interfere with goal-directed control, we developed a novel behavioral paradigm in which we combined action sequence learning through repetition with a goal-directed task component. So-called dual-target trials require the goal-directed selection of the response with the highest reward probability in a fast succession of trials with short response deadlines. Importantly, the response primed by the learned action sequence is sometimes different from that required by the goal-directed task. As expected, we found that participants learned the action sequence through repetition, as evidenced by reduced reaction times (RT) and error rates (ER), while still acting in a goal-directed manner in dual-target trials. Specifically, we found that the learned action sequence biased choices in the goal-directed task toward the sequential response, and this effect was more pronounced the better individuals had learned the sequence. Our novel task may help shed light on the acquisition of automatic behavioral patterns and habits through extensive repetition, allows to assess positive features of habitual behavior (e.g., increased response speed and reduced error rates), and importantly also the interaction of habitual and goal-directed behaviors under time pressure.

show abstract

“…In general, these results appear less easily explained by latent cause inference, because the motivational state itself would enter into the contextual inference in a way that would tend in any case to discourage instrumental responding. In any case, future work could consider the integration of model-free learning back into the current model (see also Schwober, 2021); in which case the parsing of experiences among causes would be expected to affect the progression of learning from model-based to model-free within each cause (Daw et al, 2005) whereas switching across causes might drive unlearning and renewal of habits (Miller et al, 2019; Schwöbel et al, 2021; Smith et al, 2012). Finally, other recent research has emphasized other optimizations or simplifications of model-based choice short of fully model-free habits, including temporal abstraction (Russek et al, 2017), pruning (Huys et al, 2012; Mattar and Daw, 2018) and model sharing (Glitz et al, 2022), all of which might potentially interact with latent cause inference in an extension of the current work.…”

Section: Discussionmentioning

confidence: 99%

“…That is, the basic insight of these models applies to instrumental choice: that different task contingencies may occur in different circumstances, so the organism must simultaneously figure out which tasks are active while learning to perform them. Indeed, Schwöbel et al (Schwöbel et al, 2021) recently put forward a theory nesting dual-process instrumental control (model-based learning alongside a modified model-free policy learner) underneath latent cause inference, and used it to simulate several results involving the making and breaking of habits.…”

Section: Introductionmentioning

confidence: 99%

Model based control can give rise to devaluation insensitive choice

Garrett

Allan

Daw

2022

Preprint

View full text Add to dashboard Cite

Influential recent work aims to ground psychiatric dysfunction in the brain's basic computational mechanisms. For instance, compulsive symptoms as in drug abuse have been argued to arise from an imbalance between multiple systems for instrumental learning. Computational models suggest that such multiplicity arises because the brain adaptively simplifies laborious "model-based" deliberation by sometimes relying on a cheaper, more habitual "model-free" shortcut. Support for this account comes in part from failures to appropriately change behaviour in light of new events. Notably, instrumental responding can, in some circumstances, persist despite reinforcer devaluation, perhaps reflecting control by model-free mechanisms that are driven by past reinforcement rather than knowledge of the (now devalued) outcome. However, another important line of theory, heretofore mostly studied in Pavlovian conditioning, posits a different mechanism that can also modulate behavioural change. It concerns how animals identify different rules or contingencies that may apply in different circumstances, by covertly clustering experiences into distinct groups identified with different "latent causes" or contexts. Such clustering has been used to explain the return of Pavlovian responding following extinction. Here we combine both lines of theory to investigate the consequences of latent cause inference on instrumental sensitivity to reinforcer devaluation. We show that because segregating events into different latent clusters prevents generalization between them, instrumental insensitivity to reinforcer devaluation can arise in this theory even using only model-based planning, and does not require or imply any habitual, model-free component. In simulations, these ersatz habits (like laboratory ones) emerge after overtraining, interact with contextual cues, and show preserved sensitivity to reinforcer devaluation on a separate consumption test, a standard control. While these results do not rule out a contribution of model-free learning per se, they point to a subtle and important role of state inference in instrumental learning and highlight the need for caution in using reinforcer devaluation procedures to rule in (or out) the contribution of different learning mechanisms. They also offer a new perspective on the neurocomputational substrates of drug abuse and the relevance of laboratory reinforcer devaluation procedures to this phenomenon.

show abstract

Balancing control: A Bayesian interpretation of habitual and goal-directed behavior

Cited by 19 publications

References 64 publications

Characterizing human habits in the lab

Characterizing human habits in the lab

Interaction between habits as action sequences and goal-directed behavior under time pressure

Model based control can give rise to devaluation insensitive choice

Contact Info

Product

Resources

About