Contrast Is in the Eye of the Beholder: Infelicitous Beat Gesture Increases Cognitive Load During Online Spoken Discourse Comprehension

Abstract

We investigated how two cues to contrast-beat gesture and contrastive pitch accenting-affect comprehenders' cognitive load during processing of spoken referring expressions. In two visualworld experiments, we orthogonally manipulated the presence of these cues and their felicity, or fit, with the local (sentence-level) referential context in critical referring expressions while comprehenders' task-evoked pupillary responses (TEPRs) were examined. In Experiment 1, beat gesture and contrastive accenting always matched the referential context of filler referring expressions and were therefore relatively felicitous on the global (experiment) level, whereas in Experiment 2, beat gesture and contrastive accenting never fit the referential context of filler referring expressions and were therefore infelicitous on the global level. The results revealed that both beat gesture and contrastive accenting increased comprehenders' cognitive load. For beat gesture, this increase in cognitive load was driven by both local and global infelicity. For contrastive accenting, this increase in cognitive load was unaffected when cues were globally felicitous but exacerbated when cues were globally infelicitous. Together, these results suggest that comprehenders' cognitive resources are taxed by processing infelicitous use of beat gesture and contrastive accenting to convey contrast on both the local and global levels.

Eye see what you're saying: Contrastive use of beat gesture and pitch accent affects online interpretation of spoken discourse.

“…Furthermore, we predicted that, when both the color and shape words differed between context and critical sentences (i.e., color- + shape-difference, blue triangle → red square ), the presence of locally infelicitous beat gesture and/or contrastive accenting on the color word would misleadingly suggest a referent differing from the context referent in color but not shape ( red triangle; color-contrast), hindering rather than facilitating reference resolution. By allowing us to investigate the independent and interactive influences of beat gesture and contrastive accenting on online spoken language processing, the current work differs from previous research examining the influence of contrastive accenting alone on online spoken language processing (Ito et al, 2012; Ito & Speer, 2008; Kurumada et al, 2014; Watson et al, 2008; Weber et al, 2006) and from research investigating the influences of beat gesture and contrastive accenting on offline measures of spoken language processing, such as discourse memory (Kushch & Prieto, 2016; Llanes-Coromina et al, 2018; Morett & Fraundorf, 2019), and complements our work examining online interactive effects of beat gesture and contrastive accenting on cognitive load (Morett et al, 2020).…”

Working memory relates to individual differences in speech category learning: Insights from computational modeling and pupillometry

“…By late training, differences were robust, wherein individuals with higher WM experienced a greater reduction in the overall size and rate of dilation on correct trials and had a greater difference between correct and incorrect trials in late training than individuals with lower WM. The reduction in the stimulus-evoked pupillary response across training on trials that were categorized correctly likely reflects the reduced cognitive effort (Kuchinsky et al, 2014;Morett et al, 2020;Winn et al, 2015) required for the reflexive learning system when categories are optimally learned using procedural-based learning strategies . Further, it is possible that the rate of the pupillary response over time (i.e., linear term) on correct trials may reflect the switch to using the reflexive learning system.…”

Understanding how gestures are produced and perceived.

“…For example, adult speakers who are restricted from gesturing are more disfluent (Morsella & Krauss, 2004;Rauscher et al, 1996), and they do not express as much information in their communicative signal (Driskell & Radtke, 2003), although this is not always the case (Hoetjes et al, 2014). Similarly, observers who do not have access to gesture expend more resources to access information (Morett et al, 2020), and they do not glean as much information as observers who view gestures (Goldin-Meadow & Sandhofer, 1999). Thus, gestures are functional behaviors that have positive consequences for producers and perceivers.…”