Patients with hippocampal amnesia successfully integrate gesture and speech

Abstract

During conversation, people integrate information from co-speech hand gestures with information in spoken language. For example, after hearing the sentence, "A piece of the log flew up and hit Carl in the face" while viewing a gesture directed at the nose, people tend to later report that the log hit Carl in the nose (information only in gesture) rather than in the face (information in speech). The cognitive and neural mechanisms that support the integration of gesture with speech are unclear. One possibility is that the hippocampus - known for its role in relational memory and information integration - is necessary for integrating gesture and speech. To test this possibility, we examined how patients with hippocampal amnesia and healthy and brain-damaged comparison participants express information from gesture in a narrative retelling task. Participants watched videos of an experimenter telling narratives that included hand gestures that contained supplementary information. Participants were asked to retell the narratives and their spoken retellings were assessed for the presence of information from gesture. For features that had been accompanied by supplementary gesture, patients with amnesia retold fewer of these features overall and fewer retellings that matched the speech from the narrative. Yet their retellings included features that contained information that had been present uniquely in gesture in amounts that were not reliably different from comparison groups. Thus, a functioning hippocampus is not necessary for gesture-speech integration over short timescales. Providing unique information in gesture may enhance communication for individuals with declarative memory impairment, possibly via non-declarative memory mechanisms.

Evidence of Audience Design in Amnesia: Adaptation in Gesture but Not Speech

“…Similarly, in collaborative referential barrier games, patients with amnesia mark common ground status in gesture via gesture height [ 44 ] but inconsistently in speech, arriving at concise shared labels but not marking common ground consistently with definite references [ 30 ]. A growing body of work suggests that common ground is not supported by a single memory system [ 62 ] and that gesture use may be supported by the declarative memory system for some tasks [ 34 ] while leveraging non-declarative memory in others [ 40 , 42 , 43 , 66 ]. Contributions of particular memory systems may depend on the task demands.…”

“…Indeed, emerging evidence suggests that gesture may be supported by nondeclarative memory and intact neural motor networks; the gestures of patients with Parkinson’s Disease, who show the oppositive pattern of memory impairments to amnesia (intact declarative memory but impaired procedural memory due to basal ganglia dysfunction), do not reflect their prior motor and perceptual experiences [ 40 ], and patients with Parkinson’s Disease produce fewer manner and first-person perspective gestures of high motion actions [ 41 ]. Similarly, despite severe declarative memory impairments, patients with amnesia benefit from gesture in their comprehension of narratives [ 42 ] and recognition memory for novel words [ 43 ]. Further, although the speech of patients with amnesia does not reflect communal knowledge, the gestures they produce do seem to be sensitive to common ground status; like neurotypical peers, patients with amnesia adapt gesture height, producing fewer visible gestures above the barrier over the course of a collaborative referencing game as common ground accrued with their partner [ 44 ].…”

Gesture height reflects common ground status even in patients with amnesia

“…Growing evidence for a link between gesture and non-declarative memory has been observed in other paradigms (Hilverman et al, 2018; Hilverman, Clough, Duff, & Cook, 2018; Ianì & Bucciarelli, 2017; Klooster et al, 2015). This link is bolstered by the similarities between the properties of non-declarative memory and gesture.…”

The Role of Gesture in Communication and Cognition: Implications for Understanding and Treating Neurogenic Communication Disorders

“…Indeed, the gestures of people with Parkinson's disease, which affects the basal ganglia system supporting non-declarative, or procedural memory, do not reflect their prior experiences performing a motor task (Klooster et al, 2015). Gestures themselves may also be considered implicit in that both speakers and listeners rarely consciously attend to them, and yet, both healthy adults and people with amnesia integrate information from co-speech gesture into their narrative retellings of a story, suggesting that gesture-speech integration does not depend on the hippocampal declarative memory system (Hilverman et al, 2018a). Furthermore, gesture often reflects implicit transitional knowledge states by communicating information that is not yet verbally accessible (Goldin-Meadow et al, 1993).…”