Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling

Shahar, Nitzan; Hauser, Tobias U.; Moutoussis, Michael; Moran, Rani; Keramati, Mehdi; Dolan, Raymond J.

doi:10.1371/journal.pcbi.1006803

Cited by 121 publications

(193 citation statements)

References 54 publications

Supporting

Mentioning

171

Contrasting

Order By: Relevance

“…As a reference, we simulated correct model-based agents, based on the hybrid model proposed by Daw et al with w = 1. 7 Consistent with recent work by Sharar et al, 26 even when agents have a w equal to exactly 1, used the correct model of the task, and performed 1 000 trials, the recovered w parameters were not always precisely 1 ( Figure 2I). This is expected, because parameter recovery is noisy and in the standard specification of the hybrid model w cannot be greater than 1, thus any error was an underestimate of w.…”

Section: Model-based Learning Can Be Confused With Model-free Learningsupporting

confidence: 85%

Humans are primarily model-based learners in the two-stage task

Hare

2019

Preprint

View full text Add to dashboard Cite

1Distinct model-free and model-based learning processes are thought to drive both typical and 2 dysfunctional behaviors. Data from two-stage decision tasks have seemingly shown that human 3 behavior is driven by both processes operating in parallel. In this study, however, we show that 4 more detailed task instructions lead participants to make primarily model-based choices that show 5 little, if any, model-free influence. We also demonstrate that the standard methods of analyzing 6 the two-stage task may falsely classify purely model-based agents that misunderstood the task as 7 hybrid model-based/model-free actors. Furthermore, we found evidence that many participants 8 do misunderstand the task in important ways. Overall, we argue that humans formulate a wide 9 variety of learning models. Consequently, the simple dichotomy of model-free versus model-based 10 learning is inadequate to explain behavior in the two-stage task and connections between reward 11 learning, habit formation, and compulsivity. 12 Introduction 13Once upon a time, we set out to investigate how stimulus presentation influences habitual learning in 14 humans [1]. Habits are thought to be learned via model-free learning [2], a strategy that operates by 15 strengthening or weakening associations between stimuli and actions, depending on whether the action 16 is followed by a reward or not [3]. Conversely, another strategy known as model-based learning generates 17 goal-directed behavior [2], and may even protect against habit formation [4]. Model-based learning 18 selects actions by computing action values at decision time based on a model of the environment. 19 Two-stage learning tasks ( Figure 1A) have frequently been used to dissociate model-free and model-20 based influences on choice behavior Therefore, to begin to address our questions about habits, we designed an experiment employing a 22 two-stage learning task. However, when we examined our results in detail, some of the findings did not 23 make sense at all [1]. For example, we observed negative effects of reward that could not be explained 24 by model-free or model-based learning processes. 25 After a series of analyses on choice data from human participants and simulated agents, we came to 26 the conclusion that the human participants in our sample must have been confused about how the task 27 worked. Inspired by a version of the task adapted for children [18], we modified the task instructions 28 to tell participants a story that included causes and effects within a physical system, rather than 29 give them a simple set of abstract symbols and numerical probabilities ( Figure 1B-D). Such story-like 30 instructions seemed to work well in previous studies e.g. [18, 19], and therefore, we predicted that 31 our improved instructions would alleviate participants' confusion. What we did not predict was that 32 the new instructions would eliminate nearly all evidence of model-free learning. These results left us 33 even more confused than before. 34 Our results puzzled us bec...

show abstract

Section: Model-based Learning Can Be Confused With Model-free Learningsupporting

confidence: 85%

Humans are primarily model-based learners in the two-stage task

Hare

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…On half the trials, the SS reward was available immediately (now condition), whereas on the other half of the trials, the SS reward was available only after a 30d delay (not now condition). In the now condition, the SS reward consisted of $10 available immediately and LL rewards consisted of all combinations of fourteen reward amounts (10.1, 10.2, 10.5, 11,12,15,18,20,30,40,70, 100, 130, 150 dollars) and seven delays (1,3,5,8,14,30,60 days). Trials for the not now condition where identical, with the exception that an additional delay of 30 days was added to both options, such that in not now trials, the SS reward was always associated with a 30 day delay, and LL reward delays ranged from 31 to 91 days.…”

Section: Methodsmentioning

confidence: 99%

“…In contrast, in perceptual decision-making, sequential sampling models such as the drift diffusion model (DDM) that not only account for the observed choices but also for the full reaction time distributions have a long tradition [8][9][10] . Recent work in reinforcement learning [11][12][13][14] intertemporal 15,16 and simple value-based choice [17][18][19][20] has shown that sequential sampling models can be successfully applied in these domains.…”

Section: Introductionmentioning

confidence: 99%

“…The application of sequential sampling models such as the DDM has several potential advantages over traditional softmax 6 choice rules. First, including reaction time data during model estimation may improve both the reliability of the estimated parameters 11 and parameter recovery 12 , thereby leading to more robust estimates. Second, taking into account the full reaction time distributions can reveal additional information regarding the dynamics of decision processes 13,14 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The drift diffusion model as the choice rule in inter-temporal and risky choice: a case study in medial orbitofrontal cortex lesion patients and controls

Peters

D’Esposito²

2019

Preprint

View full text Add to dashboard Cite

AbstractSequential sampling models such as the drift diffusion model have a long tradition in research on perceptual decision-making, but mounting evidence suggests that these models can account for response time distributions that arise during reinforcement learning and value-based decision-making. Building on this previous work, we implemented the drift diffusion model as the choice rule in inter-temporal choice (temporal discounting) and risky choice (probability discounting) using a hierarchical Bayesian estimation scheme. We validated our approach in data from nine patients with focal lesions to the ventromedial prefrontal cortex / medial orbitofrontal cortex (vmPFC/mOFC) and nineteen age- and education-matched controls. Choice model parameters estimated via standard softmax action selection were reliably reproduced using the drift diffusion model as the choice rule, both for temporal discounting and risky choice. Model comparison revealed that, for both tasks, the data were best accounted for by a variant of the drift diffusion model including a non-linear mapping from value-differences to trial-wise drift rates. Posterior predictive checks of the winning models revealed a reasonably good fit to individual participants reaction time distributions. We then applied this modeling framework and 1) reproduced our previous results regarding temporal discounting in vmPFC/mOFC patients and 2) showed in a previously unpublished data set on risky choice that vmPFC/mOFC patients exhibit increased risk-taking relative to controls. Analyses of diffusion model parameters revealed that vmPFC/mOFC damage abolished neither value sensitivity nor asymptote of the drift rate. Rather, it substantially increased non-decision times and reduced response caution during risky choice. Our results highlight that novel insights can be gained from applying sequential sampling models in studies of inter-temporal and risky decision-making in cognitive neuroscience.

show abstract

“…This observation is compatible with a predominantly presynaptic effect of Haloperidol in the present study, leading to an overall increase in striatal dopaminergic signaling. Importantly, we extend previous pharmacological studies by applying a modeling framework for temporal discounting data that is based on a combination of standard discounting models with the drift diffusion model (DDM) 30–33 , allowing us to comprehensively examine drug effects on RT components related to both valuation and non-valuation related processes.…”

Section: Introductionmentioning

confidence: 99%

Dopaminergic modulation of human inter-temporal choice: a diffusion model analysis using the D2-receptor-antagonist haloperidol

Wagner

Clos

Sommer

et al. 2020

Preprint

View full text Add to dashboard Cite

14The neurotransmitter dopamine is implicated in diverse functions, including reward 15 processing, reinforcement learning and cognitive control. The tendency to discount future 16 rewards in value over time has long been discussed in the context of potential dopaminergic 17 modulation. Here we examined the effect of a single dose of the D2 receptor antagonist 18 Haloperidol (2mg) on temporal discounting. Our approach extends previous human 19 pharmacological studies in two ways. First, we applied state-of-the-art computational 20 modeling based on the drift diffusion model to comprehensively examine choice dynamics. 21Second, we examined dopaminergic modulation of reward magnitude effects on temporal 22 discounting. Drift diffusion modeling revealed reduced temporal discounting and substantially 23 faster non-decision times under Haloperidol. Temporal discounting was substantially 24 increased for low vs. high reward magnitudes, but this magnitude effect was largely 25 unaffected by Haloperidol. These results were corroborated by model-free analyses as well as 26 modeling via more standard approaches using softmax action selection. We previously 27 reported elevated caudate activation under Haloperidol in this sample of participants, 28 supporting the idea that Haloperidol elevated dopamine neurotransmission, e.g. by blocking 29 inhibitory feedback via presynaptic D2 autoreceptors. The present modeling results show that 30 during inter-temporal choice, this leads to attenuated temporal discounting and increased 31 response vigor (shorter non-decision times).32 33planning is thought to rely on anticipatory mechanisms that compute future outcomes based 68 on an environmental state-space, whereas such explicit forward planning is not required for 69 model-free RL. While one study found no association between model-based RL and temporal 70 discounting 19 , another large-scale online study observed the predicted positive association: 71 participants with a stronger contribution of model-based planning to RL also showed reduced 72 temporal discounting 20 . However, in contrast to temporal discounting, where (as mentioned 73 above) elevation of DA levels via the catecholamine precursor L-DOPA lead to more 74 impulsive decision-making and steeper discounting 13 (though results in a larger sample were 75 considerably more mixed 14 ), L-DOPA instead increased reliance on model-based RL in 76 controls 21 and Parkinson's disease patients 22 . However, a recent study in a substantially larger 77 sample (n=65) did not replicate this overall effect of L-DOPA on model-based control 23 . 78Rather, in this study the effect of L-DOPA was restricted to a subset of participants with high 79 working memory capacity. 80Several task-related and contextual factors affect the degree of temporal 81 discounting 1,24 . One well-replicated behavioral effect, the magnitude effect, refers to the 82 observation that the rate of temporal discounting decreases with increasing reward 83 magnitude 25 . In humans, this effect depends on lateral p...

show abstract

Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling

Cited by 121 publications

References 54 publications

Humans are primarily model-based learners in the two-stage task

Humans are primarily model-based learners in the two-stage task

The drift diffusion model as the choice rule in inter-temporal and risky choice: a case study in medial orbitofrontal cortex lesion patients and controls

Dopaminergic modulation of human inter-temporal choice: a diffusion model analysis using the D2-receptor-antagonist haloperidol

Contact Info

Product

Resources

About