Cognitive demand is thought to modulate two often used, but rarely combined, measures: pupil size and neural alpha (8–12 Hz) oscillatory power. However, it is unclear whether these two measures capture cognitive demand in a similar way under complex audio-visual task conditions. Here we recorded pupil size and neural alpha power (using electroencephalography), while human participants of both sexes concurrently performed a visual multiple object-tracking task and an auditory gap-detection task. Difficulties of the two tasks were manipulated independent of each other. Participants’ performance decreased in accuracy and speed with increasing cognitive demand. Pupil size increased with increasing difficulty for both the auditory and the visual task. In contrast, alpha power showed diverging neural dynamics: Parietal alpha power decreased with increasing difficulty in the visual task, but not with increasing difficulty in the auditory task. Furthermore, independent of task difficulty, within-participant trial-by-trial fluctuations in pupil size were negatively correlated with alpha power. Difficulty-induced changes in pupil size and alpha power, however, did not correlate, which is consistent with their different cognitive-demand sensitivities. Overall, the current study demonstrates that the dynamics of the neurophysiological indices of cognitive demand and associated effort are multi-faceted and potentially modality-dependent under complex audio-visual task conditions.SIGNIFICANCE STATEMENTPupil size and oscillatory alpha power are associated with cognitive demand and effort, but their relative sensitivity under complex audio-visual task conditions is unclear as is the extent to which they share underlying mechanisms. Using an audio-visual dual-task paradigm, we show that pupil size increases with increasing cognitive demands for both audition and vision. In contrast, changes in oscillatory alpha power depend on the respective task demands: Parietal alpha power decreases with visual demand but not with auditory task demand. Hence, pupil size and alpha power show different sensitivity to cognitive demands, perhaps suggesting partly different underlying neural mechanisms.

Hearing loss is often asymmetric such that hearing thresholds differ substantially between the two ears. The extreme case of such asymmetric hearing is single-sided deafness. A unilateral cochlear implant (CI) on the more severely impaired ear is an effective treatment to restore hearing. The interactive effects of unilateral acoustic degradation and spatial attention to one sound source in multitalker situations are at present unclear. Here, we simulated some features of listening with a unilateral CI in young, normal-hearing listeners ( N =  22) who were presented with 8-band noise-vocoded speech to one ear and intact speech to the other ear. Neural responses were recorded in the electroencephalogram to obtain the spectrotemporal response function to speech. Listeners made more mistakes when answering questions about vocoded (vs. intact) attended speech. At the neural level, we asked how unilateral acoustic degradation would impact the attention-induced amplification of tracking target versus distracting speech. Interestingly, unilateral degradation did not per se reduce the attention-induced amplification but instead delayed it in time: Speech encoding accuracy, modelled on the basis of the spectrotemporal response function, was significantly enhanced for attended versus ignored intact speech at earlier neural response latencies (<∼250 ms). This attentional enhancement was not absent but delayed for vocoded speech. These findings suggest that attentional selection of unilateral, degraded speech is feasible but induces delayed neural separation of competing speech, which might explain listening challenges experienced by unilateral CI users.

Hearing loss is often asymmetric, such that hearing thresholds differ substantially between the two ears. The extreme case of such asymmetric hearing is single-sided deafness. A unilateral cochlear implant (CI) on the more severely impaired ear is an effective treatment to restore hearing. The neuro-cognitive cost of listening with a unilateral CI in multi-talker situations is at present unclear. Here, we simulated listening with a unilateral CI in young, normal-hearing listeners (N = 22) who were presented with 8-band noise-vocoded speech to one ear and intact speech to the other ear. Neural responses were recorded in the electroencephalogram (EEG) to obtain the spectro-temporal response function (sTRF) to speech. Listeners made more mistakes when answering questions about vocoded (versus intact) attended speech, indicating the behavioural cost of attending to spectrally degraded speech. At the neural level, we asked how unilateral acoustic degradation would impact the attention-induced amplification of tracking target versus distracting speech. Interestingly, unilateral degradation did not per se reduce the attention-induced amplification but instead delayed in time: Speech encoding accuracy, modelled on the basis of the sTRF, was significantly enhanced for attended versus ignored intact speech at earlier neural response latencies (<~250 ms). Notably, this attentional enhancement was not absent but delayed for vocoded speech. These findings suggest that attentional selection of unilateral, degraded speech is feasible but comes at the cost of delayed neural separation of competing speech, which might explain listening challenges experienced by unilateral CI users.

Cognitive demand is thought to modulate two often used, but rarely combined, measures: pupil size and neural alpha (8-12 Hz) oscillatory power. However, it is unclear whether these two measures capture cognitive demand in a similar way, and whether a measured cognitive demand response in pupil size or brain activity is dependent on sensory modality by which the demand is induced. Here we recorded pupil size and neural alpha power (using electroencephalography), while human participants of both sexes concurrently performed a visual multiple object-tracking task and an auditory gap-detection task. Difficulties of the two tasks were manipulated independent of each other. Participants' performance decreased in accuracy and speed with increasing cognitive demand. Pupil size increased with increasing demand for both the auditory and the visual task. Alpha power emerging from two cortical sources showed counteracting neural dynamics: Alpha power in visual cortex decreased with increasing demand in the visual task, whereas alpha power in parietal cortex increased with increasing demand in the auditory task. Furthermore, demand-related changes in pupil size and alpha power were uncorrelated, suggesting that they index different aspects of cognitive demand. Overall, the current study demonstrates that the dynamics of the neurophysiological indices of cognitive demand and associated effort are multi-faceted and potentially modality-dependent under complex audio-visual task conditions.

Motivation is crucial for a person to listen under acoustically demanding conditions. Previous work has demonstrated pupil-linked arousal to be sensitive to both, listening demands and motivational state during speech listening. It is less clear how motivational state affects the temporal evolution of the pupil size and its relation to subsequent behavior. Here, we used an auditory gap-detection task (N=33) to study the joint impact of listening demand and motivational state on the pupil-size response and examine its temporal evolution. Task difficulty and a listener's motivational state were orthogonally manipulated through changes in gap duration and monetary-reward prospect, respectively. We show that participants' performance decreased with task difficulty, but that reward prospect enhanced performance under hard listening conditions. Pupil size increased with both increased task difficulty and higher reward prospect, and this reward-prospect effect was largest under difficult listening conditions. Moreover, pupil-size time courses differed between detected and missed gaps, suggesting that the pupil response indicates upcoming behavioral performance. Larger pre-gap pupil size was further associated with faster response times on a trial-by-trial within-participant level. Our results reiterate the utility of pupil size as an objective and temporally sensitive measure in audiology. However, such assessments of cognitive-resource recruitment need to consider the individual's motivational state.