Intermittent Absence of Control during Reinforcement Learning Interferes with Pavlovian Bias in Action Selection

Abstract

The ability to control the occurrence of rewarding and punishing events is crucial for our well-being. Two ways to optimize performance are to follow heuristics like Pavlovian biases to approach reward and avoid loss or to rely more on slowly accumulated stimulus–action associations. Although reduced control over outcomes has been linked to suboptimal decision-making in clinical conditions associated with learned helplessness, it is unclear how uncontrollability of the environment is related to the arbitration between different response strategies. This study directly tested whether a behavioral manipulation designed to induce learned helplessness in healthy adults (intermittent loss of control over feedback in a reinforcement learning task; “yoking”) would modulate the magnitude of Pavlovian bias and the neurophysiological signature of cognitive control (frontal midline theta power) in healthy adults. Using statistical analysis and computational modeling of behavioral data and electroencephalographic signals, we found stronger Pavlovian influences and alterations in frontal theta activity in the yoked group. However, these effects were not accompanied by reduced performance in experimental blocks with regained control, indicating that our behavioral manipulation was not potent enough for inducing helplessness and impaired coping ability with task demands. We conclude that the level of contingency between instrumental choices and rewards/punishments modulates Pavlovian bias during value-based decision-making, probably via interfering with the implementation of cognitive control. These findings might have implications for understanding the mechanisms underlying helplessness in various psychiatric conditions.

Transcranial direct current stimulation above the medial prefrontal cortex facilitates decision-making following periods of low outcome controllability

“…Our primary interest was to look for potential group differences in the temporal evolution of the PB parameter π, but we also extracted parameters representing randomness of choice (temperature; β), learning rate (α) and the general tendency to initiate actions (Go-bias; b go ). This approach is identical to the one described in detail in our previous study [19], with the exception that we did not include single-trial FMθ power in the current study. Therefore, we refer the reader to that publication for details about the models and parameter estimation technique.…”

“…Relative to our earlier version of the task [19], we implemented only two experimental blocks but doubled their length to consist of 160 trials (4 cards x 40 repetitions). Participants had to collect points by learning whether to respond (Go: "pick up") or not (NoGo: "leave on the table") to each card.…”

“…Therefore, we calculated the Pavlovian performance index (PPI) as the mean of two measures, "Reward-based invigoration" (the ratio of Go responses on win trials/total number of Go) and "Punishment-based suppression" (the number of NoGo responses on avoid trials/total number of NoGo). These indices represent the likelihood to initiate actions towards rewards and inhibit responses when facing potential loss, respectively [17,19]. We also calculated response accuracy as the ratio of correct responses for each block and card type.…”

“…In these situations, "willpower" [5] or cognitive control (CC) linked to the dorsal anterior cingulate cortex (dACC) was proposed as a mechanism for suppressing maladaptive PB [16][17][18]. To provide a more direct link between LH, Pavlovian response tendencies, CC and mPFC activity, we have recently implemented a "yoking" procedure as a translational approach to induce a state resembling LH in healthy humans [19]. In line with our expectations, yoking led to stronger PB by the end of our reinforcement learning task, and this experimental manipulation also interfered with frontal midline theta power (FMθ), an electrophysiological marker of CC, putatively generated in the dACC [16].…”

“…The aims of the current study were twofold. First, we modified our yoking protocol [19] to intensify its behavioral consequences: We implemented longer experimental blocks to facilitate learning under stable response-feedback contingency (control groups), and to produce stronger yoking effects under diminished outcome controllability. We expected to find enhanced PB in yoked participants receiving sham brain stimulation, which would align well with results on the effects of reduced outcome controllability on Pavlovian-instrumental interactions [2,19].…”