The role of uncertainty in attentional and choice exploration

Abstract

The exploitation-exploration (EE) trade-off describes how, when making a decision, an organism must often choose between a safe alternative with a known pay-off, and one or more riskier alternatives with uncertain pay-offs. Recently, the concept of the EE trade-off has been extended to the examination of how organisms distribute limited attentional resources between several stimuli. This work suggests that when the rules governing the environment are certain, participants learn to "exploit" by attending preferentially to cues that provide the most information about upcoming events. However, when the rules are uncertain, people "explore" by increasing their attention to all cues that may provide information to help in predicting upcoming events. In the current study, we examine how uncertainty affects the EE trade-off in attention using a contextual two-armed bandit task, where participants explore with both their attention and their choice behavior. We find evidence for an influence of uncertainty on the EE trade-off in both choice and attention. These findings provide support for the idea of an EE trade-off in attention, and that uncertainty is a primary motivator for exploration in both choice and attentional allocation.

When Is More Uncertainty Better? A Model of Uncertainty Regulation and Effectiveness

“…This alignment determines whether individuals engage in behaviors that increase or decrease uncertainty. We draw on literature concerning the exploration-exploitation dilemma in reinforcement learning (Gershman, 2018b;Gottlieb & Oudeyer, 2018;Kaelbling, Littman, & Moore, 1996;Walker, Luque, Le Pelley, & Beesley, 2019) to distinguish opening behaviors that generate further uncertainty and opportunities for learning from closing behaviors that reduce uncertainty and rely on existing knowledge. This literature provides detailed accounts of the psychological processes that determine whether an individual exploits existing knowledge to obtain immediate rewards or explores uncertain options for possible long-term gains (Schulz & Gershman, 2019;Wilson, Geana, White, Ludvig, & Cohen, 2014).…”

“…They suggested that uncertainty creates a heightened vigilance for new information that is grounded in fundamental neural processing. Walker et al (2019) reported that people widen their attention span when they are uncertain about a reward, paying attention to a greater number of cues in their environment. Attention directs cognitive resources towards uncertainty and initiates an evaluation of whether there is an opportunity to learn and expand knowledge or to protect and take advantage of existing knowledge (Beesley et al, 2015).…”

An examination of active inference in autistic adults using immersive virtual reality.

“…In this study, participants displayed subtle, context-sensitive adjustments in visual sampling behaviour that were qualitatively consistent with optimal Bayes control. For instance, when conditions were more uncertain, individuals appeared to rely less on prior information and more on incoming, stimulus-driven attentional cues 34,35 . This was illustrated in our gaze data, where the tendency to track expected balls more closely than unexpected ones was reduced under volatile conditions (Figure 4).…”

“…Optimal actions rest on dynamic adjustments in the sampling and weighting of sensory information 13 , with physical actions used to fulfil predictions and/or reduce their uncertainty 14, 17,18 . For instance, when uncertainty in prior expectations and/or environmental stability increases (e.g., during volatile task conditions), individuals tend to rely more heavily on incoming sensory feedback, and will adjust their visual search strategies accordingly 29,34,35 . Alternatively, when sensory information is more uncertain, perhaps due to visual occlusion or deceptive opponents, more emphasis will be placed on longstanding prior expectations and 'top-down' attentional processes 13,36 .…”

Protection from uncertainty in the exploration/exploitation trade-off

“…The current paper combines the learned predictiveness design with the multi-armed bandit task to produce a two-armed contextual bandit task (Schulz, Konstantinidis, & Speekenbrink, 2018;Walker, Luque, Le Pelley, & Beesley, 2019). The contextual bandit task is similar to the traditional bandit task: on each trial there are several arms that the participant can pick, and when an arm is picked it pays out some reward value in points.…”