High verbal working memory load impairs gesture-speech integration

Abstract

While previous studies have shown the importance of visuo-spatial working memory in the processing of co-speech iconic gestures, clear evidence for a potential involvement of the verbal working memory (vWM) is currently lacking. To address this issue, participants in the present study were presented with a dual task paradigm. The main outcome variable was the performance on a Stroop-like gesture task which provides a behavioural index of gesture-speech integration. Participants performed this task under conditions of either high or low concurrent vWM load. Unlike in previous studies, the number of words to remember in the high load condition was determined by their individual verbal span rather than being fixed. Results showed reaction time costs in the form of longer reaction times for semantically incongruent gesture-speech combinations as compared to congruent combinations. However, this semantic congruency effect disappeared when the vWM load increased. This result suggests a causal involvement of verbal working memory capacity in gesture-speech integration.

Trust my gesture or my word: How do listeners choose the information channel during communication?

“…Additionally, several such studies have failed to observe an overt effect of verbal WM on gesture processing, sometimes despite a clear effect of visuospatial WM on performance. For example, Wu and Coulson (2014a) did not find speech–gesture congruency benefits to be correlated with verbal WM (cf., Kandana Arachchige et al, 2022). In contrast, research from the same group reported ERP changes to comprehension as a function of WM load (Momsen et al, 2020).…”

“…For example, Aldugom et al (2020) found that visuospatial, but not verbal, WM scores were predictive of how much individuals benefited from deictic gestures during math instructions. But what about studies that did find a relation between verbal WM and gesture processing (Kandana Arachchige et al, 2022; Momsen et al, 2020; Schubotz et al, 2021)? There are several points to note here.…”

“…First, in some studies, for example, Schubotz et al (2021), speech was distorted, thus creating greater work for the verbal system to extract the signal, which may exaggerate the role of verbal WM in comprehension. Second, many studies include dual tasking (e.g., Kandana Arachchige et al, 2022; Momsen et al, 2020), which presents a special situation. Additionally, several such studies have failed to observe an overt effect of verbal WM on gesture processing, sometimes despite a clear effect of visuospatial WM on performance.…”

Trust my gesture or my word: How do listeners choose the information channel during communication?

“…First, in some studies, for example, Schubotz et al (2021), speech was distorted, thus creating greater work for the verbal system to extract the signal, which may exaggerate the role of verbal WM in comprehension. Second, many studies include dual tasking (e.g., Kandana Arachchige et al, 2022;Momsen et al, 2020), which presents a special situation. Additionally, several such studies have failed to observe an overt effect of verbal WM on gesture processing, sometimes despite a clear effect of visuospatial WM on performance.…”