The differential roles of lexical and sublexical processing during spoken-word recognition in clear and in noise

Strauß, Antje; Wu, Tongsuo; McQueen, James M.; Scharenborg, Odette; Hintz, Florian

doi:10.1016/j.cortex.2022.02.011

Cited by 9 publications

(8 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the main effect of Gesture type, manipulated within participants, single-trial time-domain EEG data were submitted to a multi-level or "random effects" statistics approach (Strauß et al, 2022). On the first level (i.e., for each individual separately), massed independent samples t-tests were calculated to compare iconic versus control movements.…”

Section: Discussionmentioning

confidence: 99%

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension

Hintz

Khoe

Strauß

et al. 2023

Cogn Affect Behav Neurosci

Self Cite

View full text Add to dashboard Cite

In face-to-face discourse, listeners exploit cues in the input to generate predictions about upcoming words. Moreover, in addition to speech, speakers produce a multitude of visual signals, such as iconic gestures, which listeners readily integrate with incoming words. Previous studies have shown that processing of target words is facilitated when these are embedded in predictable compared to non-predictable discourses and when accompanied by iconic compared to meaningless gestures. In the present study, we investigated the interaction of both factors. We recorded electroencephalogram from 60 Dutch adults while they were watching videos of an actress producing short discourses. The stimuli consisted of an introductory and a target sentence; the latter contained a target noun. Depending on the preceding discourse, the target noun was either predictable or not. Each target noun was paired with an iconic gesture and a gesture that did not convey meaning. In both conditions, gesture presentation in the video was timed such that the gesture stroke slightly preceded the onset of the spoken target by 130 ms. Our ERP analyses revealed independent facilitatory effects for predictable discourses and iconic gestures. However, the interactive effect of both factors demonstrated that target processing (i.e., gesture-speech integration) was facilitated most when targets were part of predictable discourses and accompanied by an iconic gesture. Our results thus suggest a strong intertwinement of linguistic predictability and non-verbal gesture processing where listeners exploit predictive discourse cues to pre-activate verbal and non-verbal representations of upcoming target words.

show abstract

Section: Discussionmentioning

confidence: 99%

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension

Hintz

Khoe

Strauß

et al. 2023

Cogn Affect Behav Neurosci

Self Cite

View full text Add to dashboard Cite

show abstract

“…Previous research has shown that rate-dependent perception is a robust, low-level perceptual mechanism through which listeners take into account temporal properties (i.e., speech rate) of a context to interpret duration cues for spoken-word recognition (Bosker, 2017;Reinisch & Sjerps, 2013;Sjerps & Reinisch, 2015;Toscano & McMurray, 2015). However, previous studies have also suggested that listeners modulate their reliance on different types of information when confronted with adverse listening conditions (e.g., Derawi et al, 2022;Mattys et al, 2009;Strauss et al, 2022) suggesting that rate-dependent perception might be modulated if the signal is degraded.…”

Section: Discussionmentioning

confidence: 99%

“…Critically, it has been shown that speech perception does not always operate similarly in quiet compared with when listeners are confronted with challenging listening situations. Listeners flexibly adapt to different listening conditions and reweigh their reliance on different types of information accordingly (e.g., up-or down-weighing the use of acoustic, phonotactic, and lexical information; Derawi et al, 2022;Mattys, 2004;Mattys et al, 2009;Reese & Reinisch, 2022;Strauss et al, 2022; or the extent of considering alternative lexical candidates; Brouwer & Bradlow, 2016;McQueen & Huettig, 2012). Therefore, in order to explain the workings of speech perception in general and specific processes such as rate-dependent perception in particular, an assessment of its operation under different listening conditions is critical.…”

Section: Introductionmentioning

confidence: 99%

Encoding speech rate in challenging listening conditions: White noise and reverberation

Reinisch

Bosker

2022

Atten Percept Psychophys

View full text Add to dashboard Cite

Temporal contrasts in speech are perceived relative to the speech rate of the surrounding context. That is, following a fast context sentence, listeners interpret a given target sound as longer than following a slow context, and vice versa. This rate effect, often referred to as “rate-dependent speech perception,” has been suggested to be the result of a robust, low-level perceptual process, typically examined in quiet laboratory settings. However, speech perception often occurs in more challenging listening conditions. Therefore, we asked whether rate-dependent perception would be (partially) compromised by signal degradation relative to a clear listening condition. Specifically, we tested effects of white noise and reverberation, with the latter specifically distorting temporal information. We hypothesized that signal degradation would reduce the precision of encoding the speech rate in the context and thereby reduce the rate effect relative to a clear context. This prediction was borne out for both types of degradation in Experiment 1, where the context sentences but not the subsequent target words were degraded. However, in Experiment 2, which compared rate effects when contexts and targets were coherent in terms of signal quality, no reduction of the rate effect was found. This suggests that, when confronted with coherently degraded signals, listeners adapt to challenging listening situations, eliminating the difference between rate-dependent perception in clear and degraded conditions. Overall, the present study contributes towards understanding the consequences of different types of listening environments on the functioning of low-level perceptual processes that listeners use during speech perception.

show abstract

“…For the main effect of Gesture type, manipulated within participants, single-trial time-domain EEG data were submitted to a multi-level or 'random effects' statistics approach (see Strauß et al, 2022).…”

Section: Discussionmentioning

confidence: 99%

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension

Hintz¹,

Khoe²,

Strauß³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In face-to-face discourse, listeners exploit cues in the input to generate predictions about upcoming words. Moreover, in addition to speech, speakers produce a multitude of visual signals, such as iconic gestures, which listeners readily integrate with incoming words. Previous studies have shown that processing of target words is facilitated when these are embedded in predictable compared to non- predictable discourses and when accompanied by iconic compared to meaningless gestures. In the present study, we investigated the interaction of both factors. We recorded electroencephalogram from 60 Dutch adults while they were watching videos of an actress producing short discourses. The stimuli consisted of an introductory and a target sentence; the latter contained a target noun. Depending on the preceding discourse, the target noun was either predictable or not. Each target noun was paired with an iconic gesture and a gesture that did not convey meaning. In both conditions, gesture presentation in the video was timed such that the gesture stroke slightly preceded the onset of the spoken target by 130 ms. Our ERP analyses revealed independent facilitatory effects for predictable discourses and iconic gestures. However, the interactive effect of both factors demonstrated that target processing (i.e., gesture-speech integration) was facilitated most when targets were part of predictable discourses and accompanied by an iconic gesture. Our results thus suggest a strong intertwinement of linguistic predictability and non-verbal gesture processing where listeners exploit predictive discourse cues to pre-activate verbal and non-verbal representations of upcoming target words.

show abstract

The differential roles of lexical and sublexical processing during spoken-word recognition in clear and in noise

Cited by 9 publications

References 64 publications

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension

Encoding speech rate in challenging listening conditions: White noise and reverberation

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension

Contact Info

Product

Resources

About