Computational noise in reward-guided learning drives behavioral variability in volatile environments

Abstract

When learning the value of actions in volatile environments, humans often make seemingly irrational decisions which fail to maximize expected value. We reasoned that these 'non-greedy' decisions, instead of reflecting information seeking during choice, may be caused by computational noise in the learning of action values. Here, using reinforcement learning (RL) models of behavior and multimodal neurophysiological data, we show that the majority of non-greedy decisions stems from this learning noise. The trial-to-trial variability of sequential learning steps and their impact on behavior could be predicted both by BOLD responses to obtained rewards in the dorsal anterior cingulate cortex (dACC) and by phasic pupillary dilation -suggestive of neuromodulatory fluctuations driven by the locus coeruleus-norepinephrine (LC-NE) system. Together, these findings indicate that most of behavioral variability, rather than reflecting human exploration, is due to the limited computational precision of reward-guided learning.

Lapses in perceptual decisions reflect exploration

“…We first hypothesized that lapses might be due to a fixed amount of noise added once the decision has been made. These sources of noise could include decision noise due to imprecision (Findling et al, 2018) or motor errors (Wichmann and Hill, 2001). However, these sources should hinder decisions equally across stimulus conditions (Supplementary Fig.…”

“…Such noise could arise from errors in motor execution (e.g. finger errors, Wichmann and Hill, 2001), non-stationarities in the decision rule arising from computational imprecision (Findling et al, 2018), suboptimal weighting of choice or outcome history (Roy et al, 2018; Busse et al, 2011) or random variability added for the purpose of exploration (eg.“ ϵ -greedy” decision rules).…”

Computation noise promotes cognitive resilience to adverse conditions during decision-making

“…Beyond input noise, biological neural networks process and respond to input with a large internally generated variability 15 that is absent from artificial neural networks whose units exhibit deterministic input-state-output relations after training. This 'computation noise' has a wideranging impact on human cognition and behavior 16 -from fluctuations in the perception of weak sensory stimuli 17 to 'exploratory' behavior during reward-guided decision-making [18][19][20] . Existing research has typically considered this internal noise as a hard constraint on neural information processing systems, that the brain has evolved to cope with using efficient coding strategies [21][22][23][24] .…”

Exploration in the wild

“…There has been a considerable debate about whether or not directed exploration is required to explain human behavior 14,15 . For example, Daw and colleagues 14 have shown that a softmax strategy explains participants' choices best in a simple multiarmed bandit task.…”