Pupil dilation and response slowing distinguish deliberate explorative choices in the probabilistic learning task

Abstract

This study examined whether pupil size and response time would distinguish directed exploration from random exploration and exploitation. Eighty-nine participants performed the two-choice probabilistic learning task while their pupil size and response time were continuously recorded. Using LMM analysis, we estimated differences in the pupil size and response time between the advantageous and disadvantageous choices as a function of learning success, i.e., whether or not a participant has learned the probabilistic contingency between choices and their outcomes. We proposed that before a true value of each choice became known to a decision-maker, both advantageous and disadvantageous choices represented a random exploration of the two options with an equally uncertain outcome, whereas the same choices after learning manifested exploitation and direct exploration strategies, respectively. We found that disadvantageous choices were associated with increases both in response time and pupil size, but only after the participants had learned the choice-reward contingencies. For the pupil size, this effect was strongly amplified for those disadvantageous choices that immediately followed gains as compared to losses in the preceding choice. Pupil size modulations were evident during the behavioral choice rather than during the pretrial baseline. These findings suggest that occasional disadvantageous choices, which violate the acquired internal utility model, represent directed exploration. This exploratory strategy shifts choice priorities in favor of information seeking and its autonomic and behavioral concomitants are mainly driven by the conflict between the behavioral plan of the intended exploratory choice and its strong alternative, which has already proven to be more rewarding.

Value-driven modulation of visual perception by visual and auditory reward cues: The role of performance-contingent delivery of reward

“…Secondly, in addition to the overall heightened dilation of pupils in PC phase, we found that only in this phase value-driven modulation of pupil size was significant, and this effect was predictive of the behavioral speed modulation. Modulation of pupil responses by reward value is 10.3389/fnhum.2022.1062168 in line with a number of previous findings Braver, 2013, 2014;Massar et al, 2016;Koelewijn et al, 2018;Pietrock et al, 2019;Walsh et al, 2019) and indicates that when the delivery of reward is contingent on task performance, higher reward incentives could efficiently mobilize the processing resources, and settle an efficient relationship between the speed and accuracy of choices, effects that are also reflected in the taskevoked pupil dilatation and have been reported across motor (Naber and Murphy, 2020), perceptual (Walsh et al, 2019), and cognitive (Kozunova et al, 2022) tasks. On the other hand, the lack of value-driven modulation of pupil responses for PR cues is in line with effects reported in previous studies, where reward-driven modulations of pupil size were only found during the learning of reward associations (Anderson and Yantis, 2012) but were absent during the test phase when rewardassociations were implicit (Hammerschmidt et al, 2018).…”

Pupil size encodes uncertainty during exploration

“…The hypothesized relationship between NE and random exploration is in line with previous work showing that pupil size -as an indirect measure of NE level (4,13) -correlates with variability in the evidence accumulation process (38,39), magnitude of noise in perceptual tasks (40), and choice randomness in value-based decision making (9,(41)(42)(43)(44). These pupillometry studies suggest that pupil size could represent the computation of total uncertainty during reinforcement learning, though this has not been directly tested yet.…”

Losses resulting from deliberate exploration trigger beta oscillations in frontal cortex

“…This conflict arises between at least two simultaneously active competing internal models, or 'task sets' (Domenech et al, 2020;Koechlin, 2020) one being a predominant response tendency (exploitation), and the otherits conscious alternative (exploration). Our recent pupillometric study lends support to this assumption (Kozunova et al, 2022): we found that such explorative choices compared to exploitative ones are accompanied by larger pupil dilation and longer decision time. We speculated that this state of conflict supposedly entails an increase in the degree of processing required to make the deliberately explorative decisions.…”

“…A recent pupillometric study (Kozunova et al, 2022) revealed that advantageous choices that immediately preceded and immediately followed explorative choices significantly differed from the advantageous choices committed within the periods of continuous exploitation. This finding hints that the internal state related to exploration modulates brain activity on a scale longer than duration of one trial.…”

“…outcome of the previous trial (two levels: ‘previous loss’ and ‘previous gain’) and their interactions as the fixed effects, and Subject as a random factor: where Response time is log-transformed RT (time from stimulus onset to button press originally measured in milliseconds). We included the Previous feedback factor into the LMM model because it could affect the response time via interaction with Choice type in the same probability task (Kozunova et al, 2022).…”