Effects of acoustic and semantic cues on listening effort during native and non-native speech perception

Abstract

Relative to native listeners, non-native listeners who are immersed in a second language environment experience increased listening effort and a reduced ability to successfully perform an additional task while listening. Previous research demonstrated that listeners can exploit a variety of intelligibility-enhancing cues to cope with adverse listening conditions. However, little is known about the implications of those speech perception strategies for listening effort. The current research aims to investigate by means of pupillometry how listening effort is modulated in native and non-native listeners by the availability of semantic context and acoustic enhancements during the comprehension of spoken sentences. For this purpose, semantic plausibility and speaking style were manipulated both separately and in combination during a speech perception task in noise. The signal to noise ratio was individually adjusted for each participant in order to target 50% intelligibility level. Behavioural results indicated that native and non-native listeners were equally able to fruitfully exploit both semantic and acoustic cues to aid their comprehension. Pupil data indicated that listening effort was reduced for both groups of listeners when acoustic enhancements were available, while the presence of a plausible semantic context did not lead to a reduction in listening effort.

Listening effort is not the same as speech intelligibility score

“…In summarizing the framework of Aston-Jones and Cohen (2005), Laeng et al (2012) describe the two cognitive systems as a diffuse exploration (tonic) mode and a focused exploitation (phasic) mode, which adjusts attentional filtering to optimize performance; the phasic attentional shifts are reflected in the pupil dilation measurements in the current study. Normal language processing should be quick and efficient when the signal is clean and well-formed, and listeners with normal hearing indeed show reduced effort when listening to sentences with linguistic coherence (Winn 2016;Borghini & Hazan 2020). But when the signal is degraded either systematically via background noise (Zekveld et al 2010), spectral degradation (Winn et al 2015), divided attention (Koelewijn et al 2015; or selective word masking (Experiment 1 in the current study), momentary engagement of effort is observed in the data.…”

“…The current study is not the first to examine changes in pupil dilation linked to aspects of speech perception other than intelligibility. There are systematic differences in pupil size elicited by stimuli that vary by factors that are more subtle than all-or-none correctness, such as lexical competition (Kuchinsky et al 2013) (Hyönä et al 1995), pronoun resolution (Vogelzang et al 2016), semantic context (Winn 2016;Borghini & Hazan 2020) and lexical ambiguity (Kadem et al 2020).…”

Listening Effort Is Not the Same as Speech Intelligibility Score

“…The empirical data in the current study directly support that idea, both in terms of reconstructing phrases with missing pieces (Experiment 1) and making inferences constrained by properties of the language (Experiment 2). Normal language processing should be quick and efficient when the signal is clean and well-formed, and listeners with NH indeed show reduced effort when listening to sentences with linguistic coherence ( Borghini & Hazan, 2020 ; Winn, 2016 ). But when the signal is degraded either systematically via background noise ( Zekveld et al., 2010 ), spectral degradation ( Winn et al., 2015 ), divided attention ( Koelewijn et al., 2015 , 2017 ), or selective word masking (Experiment 1 in the current study), momentary engagement of effort is observed in the data.…”

“…The current study is not the first to examine changes in pupil dilation linked to aspects of speech perception other than intelligibility. There are systematic differences in pupil size elicited by stimuli that vary by factors that are more subtle than all-or-none correctness, such as lexical competition ( Kuchinsky et al., 2013 ), speech accentedness ( McLaughlin & Van Engen, 2020 ), speaking style (e.g., conversational versus clear; Borghini & Hazan, 2020 ), sentence structure ( Ayasse & Wingfield, 2018 ; Demberg & Sayeed, 2016 ), translation from a different language ( Hyönä et al., 1995 ), pronoun resolution ( Vogelzang et al., 2016 ), semantic context ( Borghini & Hazan, 2020 ; Winn, 2016 ), and lexical ambiguity ( Kadem et al., 2020 ). We therefore expect that different types of perceptual patterns should elicit measurable differences in listening effort reflected in pupil dilation, in a granular fashion, independent of intelligibility scores.…”

“…Essentially, this translates to: Given the range of pupil dilation that is elicited without any context, what proportion of that range is reduced by the presence of semantic context? Typically, context results in lower pupil dilation ( Winn, 2016 ), and the degree and timing of that reduction depend on various properties of the stimulus (e.g., speaking rate/style; Borghini & Hazan, 2020 ) and of the listener (e.g., hearing status: Winn, 2016 ; native language: Borghini & Hazan, 2020 ). The reference point of the peak response for low-context stimuli results in a calculation that is self-normalized for differences in language-evoked pupil reactivity across participants.…”