Thematic role assignment in patients with Broca's aphasia: Sentence–picture matching electrified

Abstract

An event-related brain potential experiment was carried out to investigate on-line thematic role assignment during sentence-picture matching in patients with Broca's aphasia. Subjects were presented with a picture that was followed by an auditory sentence. The sentence either matched the picture or mismatched the visual information depicted. Sentences differed in complexity, and ranged from simple active semantically irreversible sentences to passive semantically reversible sentences. ERPs were recorded while subjects were engaged in sentence-picture matching. In addition, reaction time and accuracy were measured. Three groups of subjects were tested: Broca patients (N = 10), non-aphasic patients with a right hemisphere (RH) lesion (N = 8), and healthy aged-matched controls (N = 15). The results of this study showed that, in neurologically unimpaired individuals, thematic role assignment in the context of visual information was an immediate process. This in contrast to patients with Broca's aphasia who demonstrated no signs of on-line sensitivity to the picture-sentence mismatches. The syntactic contribution to the thematic role assignment process seemed to be diminished given the reduction and even absence of P600 effects. Nevertheless, Broca patients showed some off-line behavioral sensitivity to the sentence-picture mismatches. The long response latencies of Broca's aphasics make it likely that off-line response strategies were used.

Different mechanisms for role relations versus verb–action congruence effects: Evidence from ERPs in picture–sentence verification

“…Mismatches between the role relations expressed in a sentence and depicted in a drawing elicited at least partially different ERP effects from those to verb-action mismatches, thereby corroborating the hypothesis that there may be functionally distinct mechanisms in mapping language to the visual context (Wassenaar & Hagoort, 2007, although see Vissers et al (2008)). …”

Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

“…In fact, when we employ continuous methods such as eye tracking and ERPs to study language processing in non-linguistic visual contexts, then a range of tasks appear suitable for providing insight into the time course and nature of language processing and into the interaction of comprehension processes with information from the nonlinguistic context. Among these are tasks in which participants act out instructions on objects (e.g., Tanenhaus, Spivey-Knowlton, Eberhard, & Sedivy, 1995), listen for comprehension (e.g., Altmann & Kamide, 1999), judge sentence veracity (e.g., Guerra & Knoeferle, 2013), and verify picture-sentence congruence (e.g., Altmann & Kamide, 1999;Knoeferle et al, 2011;Vissers, Kolk, Van de Meerendonk, & Chwilla, 2008;Wassenaar & Hagoort, 2007). Knoeferle, P. (to appear).…”

“…In one study, healthy older adults verified whether a spoken Dutch sentence (e.g., 'The tall man on [sic] this picture pushes the young woman') accurately described a previously-inspected line drawing (e.g., a man pushing a woman versus a woman pushing a man, Wassenaar & Hagoort, 2007). For active sentences, a reliably larger posterior negativity was followed by a numerically larger positivity to role relation mismatches (vs. matches) at the verb (centro-posterior from 50-450 ms; for anterior sites from ca.…”

Situated sentence processing: The coordinated interplay account and a neurobehavioral model

“…In addition, fMRI findings suggest that when participants relate information from pictures to a sentence, both typical language processing areas (the left posterior temporal gyrus) and visual-spatial processing areas (the left and right parietal areas) are activated (Carpenter, Just, Keller, Eddy, & Thulborn, 1992). Further evidence for the rapid contribution of non-linguistic visual representations to incremental sentence comprehension comes from ERP studies (e.g., Ganis, Kutas, & Sereno, 1996;Wassenaar & Hagoort, 2007;Knoeferle, Habets, et al, 2008;Knoeferle, Kutas, & Urbach, 2008). It is precisely this influence of non-linguistic, particularly visual, constraints that have not been reflected in current accounts of compositional sentence processing.…”

“…Further, by training the model on high-and low-predictability contexts, we would predict that CIANet should index an N400 when anticipated lexical material is not encountered (van Petten, Coulson, Rubin, Plante, & Parks, 1999;van Berkum et al, 2005;DeLong et al, 2005). Further, it would be interesting to extend the neural linking hypotheses to the combined early negativity/late positivity sequence elicited in response to scene-sentence relational mismatches (Wassenaar & Hagoort, 2007;Vissers, Kolk, van de Meerendonk, & Chwilla, 2008), and the N400 elicited by a verb-action mismatch (Knoeferle, Kutas, & Urbach, 2008). More systematically addressing the question of which unit subgroups in the hidden layer of the CIANet architecture correspond to which ERP components (including N400 eventually; Kutas & Hillyard, 1980, 1983van Petten & Kutas, 1990) will be essential for enabling a precise linking hypothesis to such diverse ERP findings.…”