Balancing control: a Bayesian interpretation of habitual and goal-directed behavior

Abstract

In everyday life, our behavior varies on a continuum from either automatic and habitual to deliberate and goal-directed. Recent evidence suggests that habit formation and relearning of habits operate in a context-dependent manner: Habit formation is promoted when actions are performed in a specific context, while breaking off habits is facilitated after a context change.It is an open question how one can computationally model the brain's balancing between context-specific habits and goal-directed actions. Here, we propose a hierarchical Bayesian approach for control of a partially observable Markov decision process that enables conjoint learning of habit and reward structure in a context-specific manner. In this model, habit learning corresponds to a value-free updating of priors over policies and interacts with the value-based learning of the reward structure. Importantly, the model is solely built on probabilistic inference, which effectively provides a simple explanation how the brain may balance contributions of habitual and goal-directed control. We illustrated the resulting behavior using agent-based simulated experiments, where we replicated several findings of devaluation and extinction experiments. In addition, we show how a single parameter, the so-called habitual tendency, can explain individual differences in habit learning and the balancing between habitual and goal-directed control. Finally, we discuss the relevance of the proposed model for understanding specific phenomena in substance use disorder and the potential computational role of activity in dorsolateral and dorsomedial striatum and infralimbic cortex, as reported in animal experiments.

Joint modeling of choices and reaction times based on Bayesian contextual behavioral control

“…The prior over policies on the other hand, is updated and learned based on Bayesian learning rules, which yield higher a priori probabilities for policies which have been previously chosen in this context. In our past work ( Schwöbel et al, 2021 ) we proposed to interpret this as repetition-based habit learning, as this term implements a priori tendencies to repeat policies, independent of any reward expectations. Due to the prior being over policies, i.e.…”

“…We call such an agent a “weak prior learner” as in this setting the prior learning is almost neglectable as the pseudo counts are dominated by initial values. In our previous study, we argued that the habitual tendency parameter may be used to model inter-individual differences in habit tasks ( Schwöbel et al, 2021 ). We show the effects of strong and weak prior learning on reaction times and accuracy in a sequential decision task (see Section Value-based decision making in a grid world).…”

“…Following ideas of active inference (Friston et al, 2015(Friston et al, , 2016 and the free energy principle (Friston, 2009(Friston, , 2010, an agent inverts this generative model using approximate inference based on the variational free energy. More specifically, the computational model is an extension of the recently proposed hierarchical Bayesian forward planning and habit learning model (Schwöbel et al, 2021). Using this approach, we enable an agent to infer beliefs about future states, rewards, the context, as well as probabilities of choosing different actions or sequences of actions.…”

“…As a value-based decision model we use a recently introduced Bayesian prior-based contextual control model for forward planning and goal-directed decision making ( Schwöbel et al, 2021 ). This model has two key ingredients: (i) a prior over actions or action sequences which is learned based on repetitions and independent of reward expectations, and (ii) differences in the environment are described as different contexts that need to be inferred.…”

Interaction between habits as action sequences and goal-directed behavior under time pressure

“…While the hierarchical model proposed by Dezfouli and Balleine has already been shown to reproduce experimental data of the two-stage task (Dezfouli and Balleine, 2013), it will be interesting to assess whether it can replicate key aspects of the present study. In a third approach, (Schwöbel et al, 2021) proposed a hierarchical Bayesian model that combines the idea of habit acquisition through repetition and habits as action sequences. In this model, habits are considered as precise priors over action sequences in a Bayesian integrator model, where the value-based goal-directed mode of behaviour is represented by a Bayesian likelihood function.…”