Cross-modal effects of value on perceptual acuity and stimulus encoding

Abstract

Cross-modal interactions are very common in perception. An important feature of many perceptual stimuli is their reward-predicting properties, the utilization of which is essential for adaptive behavior. What is unknown is whether reward associations in one sensory modality influence perception of stimuli in another modality. Here we show that auditory stimuli with high-reward associations increase the sensitivity of visual perception, even when sounds and reward associations are both irrelevant for the visual task. This increased sensitivity correlates with a change in stimulus representation in the visual cortex, indexed by increased multivariate decoding accuracy in simultaneously acquired functional MRI data. Univariate analysis showed that reward associations modulated responses in regions associated with multisensory processing in which the strength of modulation was a better predictor of the magnitude of the behavioral effect than the modulation in classical reward regions. Our findings demonstrate a value-driven crossmodal interaction that affects perception and stimulus encoding, with a resemblance to well-described modulatory effects of attention. We suggest that multisensory processing areas may mediate the transfer of value signals across senses.reward value | sensory discrimination | audiovisual T he world is structured such that objects or events that cause sensations in one sensory modality influence those in another modality, a mechanism that underlies the near-ubiquitous phenomenon of multisensory interaction (1). This phenomenon has been the focus of a large and burgeoning theoretical and experimental literature (1-4). One feature of environmental stimuli important for adaptive behavior is their rewarding or rewardpredicting properties. Surprisingly, we know little about how reward associations in one sensory modality influence processing in other modlities. In particular, whether an association with increased reward, known to increase the accuracy of perceptual processing within that sensory modality (5-10), can transfer between modalities for concurrently presented stimuli is unclear. This is important both because of what it reveals about the nature of cross-modal associations and because it might constitute an important perceptual mechanism in its own right. For example, one can easily imagine the importance of increased sensitivity to change in the visual scene to the concurrent sound of a tiger roaring or the mating call of a conspecific.To study cross-modal transfer of value, we designed a visual orientation discrimination task in which visual stimuli were presented concurrently with one of two arbitrary pure tones previously associated with different levels of monetary reward. Critically, these tones were task-irrelevant and bore no relationship either to the orientation of the visual stimulus or to the outcome of the trial.(No feedback was presented about the accuracy of perceptual judgments, and performance on the orientation discrimination task was not related to the payment subjects...

The Bright and Dark Sides of Performance‐Dependent Monetary Rewards: Evidence From Visual Perception Tasks

“…Once it was no longer provided, motivational engagement would return to a normal level, or even to a lower level as performance decreased more from training to posttest for the reward group. This finding was contrary to previous evidence that reward influenced visual perception even when the rewards were no longer delivered (Anderson, 2013;Anderson et al, 2011;Failing & Theeuwes, 2014Pooresmaeili et al, 2014;, even after several days (Della Libera & Chelazzi, 2009). These inconsistent findings might be due to the paradigm differences (the proactive paradigm used in the current study and the reactive paradigm used in the above studies ;Braver, 2012;Failing & Theeuwes, 2018;Pessoa, 2015).…”

“…Because there was no previous study using the same paradigm to examine the same issue, we aimed for a sample size of 50 subjects (25 subjects per group) to be comparable with, or larger than the sample sizes of previous studies that examined the influence of rewards on visual perception (Marx & Einhauser, 2015; Pooresmaeili et al, 2014; Seitz, Kim, & Watanabe, 2009; Xue et al, 2015). Based on the sensitivity analysis using G*Power software (Faul, Erdfelder, Buchner, & Lang, 2009), a sample size of 50 would allow us to detect effects ( f ) larger than 0.26 with our design of within‐between interaction of repeated measures, power = 0.95, alpha = 0.05, number of groups = 2, number of measurements = 2, correlation among measurements = 0.5, and nonsphericity correction = 1.…”

Auditory attentional selection is biased by reward cues

“…To our knowledge, two studies used auditory reward-learning to form simple stimulus-reward associations; and tested their impact on visual perception23 and attention24. Pooersmaeili and colleagues23 found that high-reward associated sounds increased visual sensitivity in an orientation discrimination task, while Anderson24 found that high-reward associated sounds interfered with the performance in a visual attention task. These seemingly contradicting results indicate that it is currently unknown how exactly the value representations in auditory domain would affect visual processes.…”

Cross-modal integration of reward value during oculomotor planning

“…Among the top-down influences on perception, the effect of reward is particularly important to motivate an agent, facilitate learning, and help the agent to behave adaptively given the limited capacity of both sensory and motor systems. It has been shown that reward acts to modulate selective attention when the subject's performance was directly linked to the monetary reward (Small et al, 2005;Mohanty et al, 2008;Engelmann et al, 2009), even when the monetary reward was no longer task-relevant (Libera and Chelazzi, 2006;Hickey and van Zoest, 2012;Pooresmaeili et al, 2014;Asutay and Västfjäll, 2016;Luque et al, 2017). Rewards may act as guiding signals for learning and optimizing specific attentional operations (Chelazzi et al, 2013a), and not only can reward increase the salience of associated stimuli (Hickey and van Zoest, 2012), but also enhance the suppression of the distractors (Della Libera and Chelazzi, 2009), change the priority maps of space (Chelazzi et al, 2014), and reduce the intrinsic neural noise in the motor and cognitive control (Manohar et al, 2015).…”