Transition from ‘model-based’ to ‘model-free’ behavioral control in addiction: Involvement of the orbitofrontal cortex and dorsolateral striatum

Abstract

Cocaine addiction is a complex and multidimensional process involving a number of behavioral and neural forms of plasticity. The behavioral transition from voluntary drug use to compulsive drug taking may be explained at the neural level by drug-induced changes in function or interaction between a flexible planning system, associated with prefrontal cortical regions, and a rigid habit system, associated with the striatum. The dichotomy between these two systems is operationalized in computational theory by positing model-based and model-free learning mechanisms, the former relying on an “internal model” of the environment and the latter on pre-computed or cached values to control behavior. In this review, we will suggest that model-free and model-based learning mechanisms appear to be differentially affected, at least in the case of psychostimulants such as cocaine, with the former being enhanced while the latter are disrupted. As a result, the behavior of long-term drug users becomes less flexible and responsive to the desirability of expected outcomes and more habitual, based on the long history of reinforcement. To support our specific proposal, we will review recent neural and behavioral evidence on the effect of psychostimulant exposure on orbitofrontal and dorsolateral striatum structure and function.

Hippocampus and subregions of the dorsal striatum respond differently to a behavioral strategy change on a spatial navigation task

“…Different cortical substrates are involved in different environmental/rewardrelated behavioral changes. For example, adjusting behavioral strategies, reversal learning, and contingency degradation are all mediated by different cortical areas (Corbit et al 2002;Hauber 2010a, 2010b;Lucantonio et al 2014;Ragozzino 2007;Schoenbaum et al 2002). Common among the evaluation of each change to the environment is the necessity of flexibly associating the outcome with the preceding action.…”

“…Several recent studies have implicated the pDMS and its related input structures mediating these sorts of rewardrelated behavioral shifts (Corbit et al 2002;Izquierdo et al 2004;Killcross and Coutureau 2003;Hauber 2010a, 2010b;Shiflett et al 2010;Yin et al 2005aYin et al , 2005b, making pDMS/orbitofrontal and pDMS/prelimbic circuits important for altering behavior when an unexpected variation occurs in the environment. Recent studies indicate that cortical areas may evaluate state changes (for reviews, see Lucantonio et al 2014;Ragozzino 2007;Torregrossa et al 2008) and pDMS may integrate information from cortical areas into appropriate actions (Kimchi and Laubach 2009;Stalnaker et al 2012).…”

“…The anatomical inputs to pDMS are from structures involved in reversal learning, strategy changing, and action-outcome learning. Interestingly, studies have implicated the pDMS in many of these specific types of learning Hauber 2010a, 2010b;Lucantonio et al 2014;Stalnaker et al 2012;Yin et al 2005aYin et al , 2005b.…”

Nucleus Accumbens Core and Shell Differentially Encode Reward-Associated Cues after Reinforcer Devaluation

“…There are several necessary processes to perform this task: (1) forming an association between a cue and an outcome, (2) registering the decreased value of the reinforcer after its devaluation, and (3) integrating the learned cue-outcome association with the decreased outcome value to direct behavior. Importantly, testing is performed under extinction, so rats must use an internal representation of the previously learned association and alter behavior based on the newly computed expected outcome (Murray and Izquierdo, 2007;Murray et al, 2007;Lucantonio et al, 2014).…”

Reward Processing in Motivational and Affective Disorders

“…The choice of action proceeds by thinking forward through the map (or tree), and considering the consequences of alternative actions (see Seymour and Dolan, 2008). This mode of control is referred to in reinforcement learning applications as model-based (Gläscher et al, 2010;Daw et al, 2011;Wunderlich et al, 2012;Smittenaar et al, 2013;Lucantonio et al, 2014), and corresponds to the definition of goal-directed behavior in animal learning as being rapidly sensitive to changes in the contingency between action and outcome, or to devaluing the outcome (Dickinson and Balleine, 1994;Balleine and Dickinson, 1998). An advantage of the modelbased approach lies in its flexibility.…”