Learning with Reinforcement: the Role of Immediate Feedback and the Internal Model of the Situation

Kozunova, Galina L.; Voronin, N. A.; Venediktov, V. V.; Строганова, Т. А.

doi:10.1007/s11055-019-00852-7

“…During the experiment, participants were seated in a comfortable armchair with a projection screen located in front of them at eye level. A modified probabilistic learning task ( Frank et al, 2004 ; Kozunova et al, 2018 ) was rendered as a computer game consisting of six blocks.…”

Section: Methodsmentioning

confidence: 99%

Losses resulting from deliberate exploration trigger beta oscillations in frontal cortex

Владимирович

¹

,

Pultsina

²

,

Tretyakova

³

et al. 2023

View full text Add to dashboard Cite

We examined the neural signature of directed exploration by contrasting MEG beta (16–30 Hz) power changes between disadvantageous and advantageous choices in the two-choice probabilistic reward task. We analyzed the choices made after the participants have learned the probabilistic contingency between choices and their outcomes, i.e., acquired the inner model of choice values. Therefore, rare disadvantageous choices might serve explorative, environment-probing purposes. The study brought two main findings. Firstly, decision making leading to disadvantageous choices took more time and evidenced greater large-scale suppression of beta oscillations than its advantageous alternative. Additional neural resources recruited during disadvantageous decisions strongly suggest their deliberately explorative nature. Secondly, an outcome of disadvantageous and advantageous choices had qualitatively different impact on feedback-related beta oscillations. After the disadvantageous choices, only losses—but not gains—were followed by late beta synchronization in frontal cortex. Our results are consistent with the role of frontal beta oscillations in the stabilization of neural representations for selected behavioral rule when explorative strategy conflicts with value-based behavior. Punishment for explorative choice being congruent with its low value in the reward history is more likely to strengthen, through punishment-related beta oscillations, the representation of exploitative choices consistent with the inner utility model.

show abstract

“…During the experiment, participants were comfortably seated in an armchair and placed their heads on the chin rest to minimize involuntary head movements. We used the modified probabilistic learning task (Frank et al, 2004;Kozunova et al, 2018), which was rendered as a computer game. On each trial, participants had to choose between two stimuli presented on the screen simultaneously.…”

Section: Methodsmentioning

confidence: 99%

Pupil dilation and response slowing distinguish deliberate explorative choices in the probabilistic learning task

Kozunova

¹

,

Sayfulina

²

,

Prokofyev

³

et al. 2022

Cogn Affect Behav Neurosci

9

22

0

View full text Add to dashboard Cite

This study examined whether pupil size and response time would distinguish directed exploration from random exploration and exploitation. Eighty-nine participants performed the two-choice probabilistic learning task while their pupil size and response time were continuously recorded. Using LMM analysis, we estimated differences in the pupil size and response time between the advantageous and disadvantageous choices as a function of learning success, i.e., whether or not a participant has learned the probabilistic contingency between choices and their outcomes. We proposed that before a true value of each choice became known to a decision-maker, both advantageous and disadvantageous choices represented a random exploration of the two options with an equally uncertain outcome, whereas the same choices after learning manifested exploitation and direct exploration strategies, respectively. We found that disadvantageous choices were associated with increases both in response time and pupil size, but only after the participants had learned the choice-reward contingencies. For the pupil size, this effect was strongly amplified for those disadvantageous choices that immediately followed gains as compared to losses in the preceding choice. Pupil size modulations were evident during the behavioral choice rather than during the pretrial baseline. These findings suggest that occasional disadvantageous choices, which violate the acquired internal utility model, represent directed exploration. This exploratory strategy shifts choice priorities in favor of information seeking and its autonomic and behavioral concomitants are mainly driven by the conflict between the behavioral plan of the intended exploratory choice and its strong alternative, which has already proven to be more rewarding.

show abstract

“…During the experiment, participants were comfortably seated in an armchair and placed their heads on the chin rest to minimize involuntary head movements. We used the modified probabilistic learning task (Frank, Seeberger, & O'Reilly, 2004;Kozunova, Voronin, Venidiktov, & Stroganova, 2018), which was rendered as a computer game. On each trial, participants had to choose between two stimuli presented on the screen simultaneously.…”

Section: Methodsmentioning

confidence: 99%

Pupil Dilation and Response Slowing Distinguish Deliberate Explorative Choices in the Probabilistic Learning Task

Kozunova

¹

,

Sayfulina

²

,

Prokofyev

³

et al. 2021

Preprint

View full text Add to dashboard Cite

This study examined whether pupil size and response time would distinguish directed exploration from random exploration and exploitation. Eighty-nine participants performed the two-choice probabilistic learning task while their pupil size and response time were continuously recorded. Using LMM analysis, we estimated differences in the pupil size and response time between the advantageous and disadvantageous choices as a function of learning success, i.e., whether or not a participant has learned the probabilistic contingency between choices and their outcomes. We proposed that before a true value of each choice became known to a decision-maker, both advantageous and disadvantageous choices represented a random exploration of the two options with an equally uncertain outcome, whereas the same choices after learning manifested exploitation and direct exploration strategies, respectively. We found that disadvantageous choices were associated with increases both in response time and pupil size, but only after the participants had learned the choice-reward contingencies. For the pupil size, this effect was strongly amplified for those disadvantageous choices that immediately followed gains as compared to losses in the preceding choice. Pupil size modulations were evident during the behavioral choice rather than during the pretrial baseline. These findings suggest that occasional disadvantageous choices, which violate the acquired internal utility model, represent directed exploration. This exploratory strategy shifts choice priorities in favor of information seeking and its autonomic and behavioral concomitants are mainly driven by the conflict between the behavioral plan of the intended exploratory choice and its strong alternative, which has already proven to be more rewarding.

show abstract

Learning with Reinforcement: the Role of Immediate Feedback and the Internal Model of the Situation

Cited by 8 publications

References 16 publications

Losses resulting from deliberate exploration trigger beta oscillations in frontal cortex

Losses resulting from deliberate exploration trigger beta oscillations in frontal cortex

Pupil dilation and response slowing distinguish deliberate explorative choices in the probabilistic learning task

Pupil Dilation and Response Slowing Distinguish Deliberate Explorative Choices in the Probabilistic Learning Task

Contact Info

Product

Resources

About