Choice-confirmation bias and gradual perseveration in human reinforcement learning

Abstract

Do we preferentially learn from outcomes that confirm our choices? This is one of the most basic, and yet consequence-bearing, questions concerning reinforcement learning. In recent years, we investigated this question in a series of studies implementing increasingly complex behavioral protocols. The learning rates fitted in experiments featuring partial or complete feedback, as well as free and forced choices, were systematically found to be consistent with a choice-confirmation bias. This result is robust across a broad range of outcome contingencies and response modalities. One of the prominent behavioral consequences of the confirmatory learning rate pattern is choice hysteresis: that is the tendency of repeating previous choices, despite contradictory evidence. As robust and replicable as they have proven to be, these findings were (legitimately) challenged by a couple of studies pointing out that a choice-confirmatory pattern of learning rates may spuriously arise from not taking into consideration an explicit choice autocorrelation term in the model. In the present study, we re-analyze data from four previously published papers (in total nine experiments; N=363), originally included in the studies demonstrating (or criticizing) the choice-confirmation bias in human participants. We fitted two models: one featured valence-specific updates (i.e., different learning rates for confirmatory and disconfirmatory outcomes) and one additionally including an explicit choice autocorrelation process (gradual perseveration). Our analysis confirms that the inclusion of the gradual perseveration process in the model significantly reduces the estimated choice-confirmation bias. However, in all considered experiments, the choice-confirmation bias remains present at the meta-analytical level, and significantly different from zero in most experiments. Our results demonstrate that the choice-confirmation bias resists the inclusion of an explicit choice autocorrelation term, thus proving to be a robust feature of human reinforcement learning. We conclude by discussing the psychological plausibility of the gradual perseveration process in the context of these behavioral paradigms and by pointing to additional computational processes that may play an important role in estimating and interpreting the computational biases under scrutiny.

Individuals with problem gambling and obsessive-compulsive disorder learn through distinct reinforcement mechanisms

“…In theory, inflexibility can also be captured by asymmetric learning for positive and negative reward prediction errors. That is, excessive learning from better-than-expected experiences and/or deficient learning from worse-than-expected experiences may result in the repetitive choice of the same option [42,43]. Consistent with the theoretical consideration, we demonstrated that, in reward-seeking decision-making, the learning rate for negative reward prediction errors was lower in patients with OCD than in healthy controls.…”

“…Yet, previous studies in computational psychiatry have often failed to detect increased perseveration in OCD and PG patients [37][38][39][40][41]. An alternative account posits that inflexibility reflects asymmetric RL, i.e., overlearning from better-than-expected outcomes and underlearning from worse-than-expected outcomes [42,43]. In formal terms, the learning rate-the extent to which new information (e.g., reward, loss, and neutral outcome) modulates future behaviour-is different for value updating from positive and negative reward prediction errors.…”

“…In formal terms, the learning rate-the extent to which new information (e.g., reward, loss, and neutral outcome) modulates future behaviour-is different for value updating from positive and negative reward prediction errors. This asymmetric learning results in excessive reinforcement and deficient devaluation of the current action, leading to inflexibility of choice [43]. However, examinations of the asymmetric RL account in OCD and PG remain sparse.…”

Examinations of Biases by Model Misspecification and Parameter Reliability of Reinforcement Learning Models

“…That is, there was a larger influence of inverse temperature, 𝛽, than the perseveration size, 𝜙, on decision-making. This finding is in accordance with that of Palminteri (2021), but as he also pointed out, the type of task used may influence which parameter has a larger impact on choices.…”

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

“…That is, as learning promotes consistent repetition of responses within a state, so too can autocorrelational effects of hysteresis producing response repetition or alternation that coincidentally aligns with rotating states. (For example, perseveration offers a more parsimonious explanation for action repetition that could otherwise be attributed to an optimistic confirmation bias (Frank et al, 2004;Sharot, 2011;Sharot et al, 2011;Thorndike, 1932Thorndike, , 1933; in RL terms, the latter could translate to an asymmetry in learning rates favoring positive over negative outcomes (Cazé & van der Meer, 2013;Daw et al, 2002;Frank et al, 2007Frank et al, , 2009Niv et al, 2012)-but at the cost of susceptibility to overfitting (relative to hysteresis) (Chambon et al, 2020;Gershman, 2016;Katahira, 2015Katahira, , 2018Palminteri, 2021;Sugawara & Katahira, 2021).) The baseline hysteresis model includes a dynamic perseveration (or alternation) bias β t (a) (cf.…”