Deciding How To Decide: Self-Control and Meta-Decision Making

Boureau, Y-Lan; Sokol‐Hessner, Peter; Daw, Nathaniel D.

doi:10.1016/j.tics.2015.08.013

Cited by 177 publications

(165 citation statements)

References 96 publications

(164 reference statements)

Supporting

Mentioning

153

Contrasting

Unclassified

Order By: Relevance

“…Our results show that humans are equipped with a much richer repertoire of strategies, than just two dichotomous systems, for coping with the complexity of real-life problems as well as with limitations in their cognitive resources. Therefore, the factors that have been shown or suggested to influence competitive combination in favor of habitual responding, such as working memory load (13), opportunity cost (27,28), stress (22), or the one we examined directly, namely time (19,29), would all be expected to favor shallower trees and thus relatively greater dominance of habits. Another recent study suggests that humans plan (18) toward goals and that targeted goals are reinforced when subjects are rewarded, resulting in higher tendency of targeting those goals in the future.…”

Section: Discussionmentioning

confidence: 99%

Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum

Keramati

Smittenaar

Dolan

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

182

203

View full text Add to dashboard Cite

Behavioral and neural evidence reveal a prospective goal-directed decision process that relies on mental simulation of the environment, and a retrospective habitual process that caches returns previously garnered from available choices. Artificial systems combine the two by simulating the environment up to some depth and then exploiting habitual values as proxies for consequences that may arise in the further future. Using a three-step task, we provide evidence that human subjects use such a normative plan-until-habit strategy, implying a spectrum of approaches that interpolates between habitual and goal-directed responding. We found that increasing time pressure led to shallower goal-directed planning, suggesting that a speed-accuracy tradeoff controls the depth of planning with deeper search leading to more accurate evaluation, at the cost of slower decision-making. We conclude that subjects integrate habit-based cached values directly into goal-directed evaluations in a normative manner.planning | habit | reinforcement learning | speed/accuracy tradeoff | tree-based evaluation B ehavioral and neural evidence suggest that the brain uses distinct goal-directed and habitual systems for decisionmaking (1-5). A goal-directed system exploits an individual's model, i.e., their knowledge of environmental dynamics, to simulate the consequences that will likely follow a choice (6) (Fig. 1A). Such evaluations, which assess a decision-tree expanding into the future to estimate the total reward, adapt flexibly to changes in environmental dynamics or the values of outcomes. Evaluating deep trees, however, is computationally expensive (in terms of time, working memory, metabolic energy, etc.) and potentially error-prone. By contrast, the habitual system simply caches the rewards received on previous trials conditional on the choice (Fig. 1C) without a representational characterization of the environment (hence being called "model-free") (6, 7). This process hinders adaptation to changes in the environment, but has advantageous computational simplicity. Previous studies show distinct behavioral and neurobiological signatures of both systems (8-18). Furthermore, consistent with the theoretical strengths and weaknesses of each system (2, 19), different experimental conditions influence the relative contributions of the two systems in controlling behavior according to their respective competencies (20)(21)(22)(23).Here, we suggest that individuals, rather than simply showing greater reliance on the more competent system in each condition, combine the relative strengths of the two systems in a normative manner by integrating habit-based cached values directly into goaldirected evaluations. Specifically, we propose that given available resources (time, working memory, etc.), individuals decide the depth k up to which they can afford full forward simulations and use cached habitual values thereafter. That is, individuals compute the value of a choice by adding the first k rewards, predicted by the explicit simulation, to the value...

show abstract

Section: Discussionmentioning

confidence: 99%

Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum

Keramati

Smittenaar

Dolan

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

182

203

View full text Add to dashboard Cite

show abstract

“…Building on previous work modeling the specification of cognitive control in terms of metadecision making [12][13][14]29,33,65,66] and reinforcement learning [33,34,[67][68][69], we have illustrated that at least some of the functions of cognitive control can be characterized using the formal framework of rational metareasoning [26] and meta-level Markov decision processes [27]. Concretely, modeling the function of cognitive control as a meta-level MDP allowed us to derive the first formal computational model of how people learn to specify continuous control signals and how these learning effects transfer to novel situations.…”

Section: Discussionmentioning

confidence: 99%

“…In broad accord with the predictions of the EVC theory, previous research has found that control specification is context-sensitive [8,9] and modulated by reward across multiple domains [10,11], such as attention, response inhibition, interference control, and task switching. While previous theories account for that fact that people's performance in these task is sensitive to reward [7,[12][13][14], it remains unclear how these dependencies arise from people's experience. Recently, it has been proposed that the underlying mechanism is associative learning [15,16].…”

Section: Introductionmentioning

confidence: 99%

Rational metareasoning and the plasticity of cognitive control

Lieder¹,

Shenhav²,

Musslick³

et al. 2017

Preprint

View full text Add to dashboard Cite

The human brain has the impressive capacity to adapt how it processes information to highlevel goals. While it is known that these cognitive control skills are malleable and can be improved through training, the underlying plasticity mechanisms are not well understood. Here, we develop and evaluate a model of how people learn when to exert cognitive control, which controlled process to use, and how much effort to exert. We derive this model from a general theory according to which the function of cognitive control is to select and configure neural pathways so as to make optimal use of finite time and limited computational resources. The central idea of our Learned Value of Control model is that people use reinforcement learning to predict the value of candidate control signals of different types and intensities based on stimulus features. This model correctly predicts the learning and transfer effects underlying the adaptive control-demanding behavior observed in an experiment on visual attention and four experiments on interference control in Stroop and Flanker paradigms. Moreover, our model explained these findings significantly better than an associative learning model and a Win-Stay Lose-Shift model. Our findings elucidate how learning and experience might shape people's ability and propensity to adaptively control their minds and behavior. We conclude by predicting under which circumstances these learning mechanisms might lead to self-control failure. Author summaryThe human brain has the impressive ability to adapt how it processes information to high level goals. While it is known that these cognitive control skills are malleable and can be improved through training, the underlying plasticity mechanisms are not well understood. Here, we derive a computational model of how people learn when to exert cognitive control, which controlled process to use, and how much effort to exert from a formal theory of the function of cognitive control. Across five experiments, we find that our model correctly predicts that people learn to adaptively regulate their attention and decision- Funding: This research was supported by the Office of Naval Research through Grant No. MURI N00014-13-1-0431 to TLG. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests:The authors have declared that no competing interests exist.making and how these learning effects transfer to novel situations. Our findings elucidate how learning and experience might shape people's ability and propensity to adaptively control their minds and behavior. We conclude by predicting under which circumstances these learning mechanisms might lead to self-control failure.

show abstract

“…The agent can learn its opportunity cost by estimating its reward rate (Boureau et al, 2015;Niv, Daw, Joel, & Dayan, 2007), and the utility of applying the strategy and its execution time can be observed. Therefore, when the reward is continuous, then it is possible to learn an efficient approximation to the VOC by learning linear predictive models of the utility of its decisions and its execution time and combining them according to…”

Section: A Rational Process Model Of Strategy Selectionmentioning

confidence: 99%

“…Despite the previous work reviewed in the following section and some recent progress on how the brain decides how to decide (Boureau, SokolHessner, & Daw, 2015) the strategy selection problem remains unsolved (Marewski & Link, 2014). Finally, while it is typically assumed that people's use of heuristics is STRATEGY SELECTION AS RATIONAL METAREASONING 4 irrational (Ariely, 2009;Marcus, 2009;Sutherland, 2013), there is increasing evidence for adaptive strategy selection (Boureau et al, 2015;Braver, 2012;Daw, Niv, & Dayan, 2005;Fum & Del Missier, 2001;Gunzelmann & Anderson, 2003;Keramati, Dezfouli, & Piray, 2011;Payne et al, 1988). This raises the additional question whether and to what extent people's strategy choices are rational.…”

mentioning

confidence: 99%

Strategy selection as rational metareasoning

Lieder¹,

Griffiths²

2016

Preprint

View full text Add to dashboard Cite

Many contemporary accounts of human reasoning assume that the mind is equipped with multiple heuristics that could be deployed to perform a given task. This raises the question of how the mind determines when to use which heuristic. To answer this question, we developed a rational model of strategy selection, based on the theory of rational metareasoning developed in the artificial intelligence literature. According to our model people learn to efficiently choose the strategy with the best cost-benefit tradeoff by learning a predictive model of each strategy's performance. We found that our model can provide a unifying explanation for classic findings from domains ranging from decisionmaking to arithmetic by capturing the variability of people's strategy choices, their dependence on task and context, and their development over time. Systematic model comparisons supported our theory, and four new experiments confirmed its distinctive predictions. Our findings suggest that people gradually learn to make increasingly more rational use of fallible heuristics. This perspective reconciles the two poles of the debate about human rationality by integrating heuristics and biases with learning and rationality.Keywords: bounded rationality; strategy selection; heuristics; meta-decisionmaking; metacognitive reinforcement learningTo succeed in life we have to solve a wide range of problems that place very different demands on us: sometimes we have to think fast and sometimes we have to think slow (cf. Kahneman, 2011). For instance, avoiding a car accident requires a split-second decision, whereas founding a successful start-up requires investing a lot of time into anticipating the future and weighting potential outcomes appropriately. No single decision mechanism works well across all situations. To meet the wide range of demands posed by different decision problems, it has been proposed that the human brain is equipped with multiple decision systems (Dolan & Dayan, 2013) and decision strategies (Payne, Bettman, & Johnson, 1988). Dual-process theories are a prominent example of this perspective (Evans & Stanovich, 2013;Evans, 2003;Kahneman, 2011). The coexistence of multiple alternative strategies is not specific to decision making. People also appear to possess multiple strategies for inference (Gigerenzer & Selten, 2002), memory (Bjorklund & Douglas, 1997), self-control (Braver, 2012), problem solving (Fum & Del Missier, 2001), and mental arithmetic (Siegler, 1999) to name just a few.The availability of multiple strategies that are applicable to the same problems raises the question how people decide when to use which strategy. The fact that so many different strategies have been observed under different circumstances shows that people's strategy choices are highly variable and contingent on the situation and the task (Beach & Mitchell, 1978;Fum & Del Missier, 2001;Payne, 1982;Payne et al., 1988). Overall, the contingency of people's strategy choices appears to be adaptive. Even though under certain circumstances p...

show abstract

Deciding How To Decide: Self-Control and Meta-Decision Making

Cited by 177 publications

References 96 publications

Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum

Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum

Rational metareasoning and the plasticity of cognitive control

Strategy selection as rational metareasoning

Contact Info

Product

Resources

About