Hesitations in story retelling as a measure of listening effort

Abstract

Hesitation phenomena in natural speech (e.g., pauses, word repetitions, or phrase repairs) vary as a function of speaker characteristics, with non-native speakers typically hesitating more than native speakers (Gilquin, 2008). Here, we examine whether hesitations can also reflect listeners’ mental effort required to comprehend native and non-native speech. We asked native English listeners to retell stories told by either a native or a non-native speaker and analyzed their hesitations using a 6-category taxonomy: filler vocalizations, filler words, filler nonspeech, self-monitoring, uptalk, and point of view switching. These story-retell behavioral data were obtained after recording their brain responses during listening to the stories using functional near-infrared spectroscopy (fNIRS) and speaker-listener interbrain synchrony technique (hyperscanning). We found a higher hesitation rate for retelling of stories produced by a non-native (11.2%) than a native speaker (7.6%), with significant increase in hesitation frequency for the former. The hesitations corroborated brain responses: Listeners had greater difficulty processing discourse in non-native English and the recruitment of additional executive resources delayed comprehension and memory processes. The analysis of hesitations is a promising approach in measuring speaker-listener communicative effort because hesitations increase as uncertainty of interpretation increases, suggesting greater demands on working memory during lexical and semantic operations.