Rational metareasoning and the plasticity of cognitive control

Abstract

The human brain has the impressive capacity to adapt how it processes information to high-level goals. While it is known that these cognitive control skills are malleable and can be improved through training, the underlying plasticity mechanisms are not well understood. Here, we develop and evaluate a model of how people learn when to exert cognitive control, which controlled process to use, and how much effort to exert. We derive this model from a general theory according to which the function of cognitive control is to select and configure neural pathways so as to make optimal use of finite time and limited computational resources. The central idea of our Learned Value of Control model is that people use reinforcement learning to predict the value of candidate control signals of different types and intensities based on stimulus features. This model correctly predicts the learning and transfer effects underlying the adaptive control-demanding behavior observed in an experiment on visual attention and four experiments on interference control in Stroop and Flanker paradigms. Moreover, our model explained these findings significantly better than an associative learning model and a Win-Stay Lose-Shift model. Our findings elucidate how learning and experience might shape people’s ability and propensity to adaptively control their minds and behavior. We conclude by predicting under which circumstances these learning mechanisms might lead to self-control failure.

Preparing for hard times: Scalp and intracranial physiological signatures of proactive cognitive control

“…It is also well established that the experience (Gratton, Coles, & Donchin, 1992) or expectation (Aarts & Roelofs, 2011) of task difficulty enhances control. This influence of reward and difficulty information on control was emphasized in theoretical accounts (Brehm & Self, 1989) and formalized in computational reinforcement learning (RL) models (Lieder, Shenhav, Musslick, & Griffiths, 2018;Silvetti, Vassena, Abrahamse, & Verguts, 2018;Verguts, Vassena, & Silvetti, 2015). Specifically, recent RL models assume that cognitive agents calculate and optimize not just expected reward (i.e., value) but a (linear) combination of reward and difficulty cost (e.g., reward − difficulty cost) in order to decide on whether to invest control or not.…”

Learning to overexert cognitive control in the Stroop task

“…amount of control allocated, and a reconfiguration cost that penalizes diverging from the most recent control signal, that is with control cost parameters set to and . Following Lieder et al (2018), the opportunity cost parameter was set to points per second which corresponds to an hourly wage of about $8/hour. The prior distribution on each weight is where and are free parameters that are shared across all weights.…”

“…We base our examination on a recently developed model that describes cognitive control allocation as the result of a cost-benefit analysis, with individuals weighing the expected payoffs for engaging control against the effort-related costs associated with doing so, to determine the overall Expected Value of Control (EVC) for a particular control allocation (Shenhav et al, 2013;Musslick et al, 2015). Building on previous models of strategy learning (Lieder et al, 2017), we recently described a set of learning algorithms that would allow someone to learn EVC through experience performing different tasks (Lieder et al, 2018). According to this Learned Value of Control (LVOC) model, people learn to predict the value of control based on features of the task environment, and they select their control allocation accordingly.…”

“…Participants came to name the color faster and more accurately for those that were rewarded than those that were not. We have shown that the LVOC model can account for such findings by decomposing a task into its component features and learning the predictive relationship between those features and future rewards (Lieder et al, 2018). For the Stroop task, each stimulus can be decomposed into two features: the color (e.g.…”

Combined model-free and model-sensitive reinforcement learning in non-human primates

“…This originally inspired ideas that their output should be combined (8). Recently, rather complex patterns of interaction have been investigated, including MB training of MF (9,10), MF control over MB calculations (11)(12)(13), the incorporation of MF values into MB calculations (14) and, of particular relevance for the present study, the creation of sophisticated, model-dependent, representations of the task that enable MF methods to work more efficiently (15), and potentially less susceptible to distraction (16). We deem these various interactions model-sensitive (MS), saving model-based for the original notion of prospective planning.…”