Situated sentence processing: The coordinated interplay account and a neurobehavioral model

Abstract

Empirical evidence demonstrating that sentence meaning is rapidly reconciled with the visual environment has been broadly construed as supporting the seamless interaction of visual and linguistic representations during situated comprehension. Based on recent behavioral and neuroscientific findings, however, we argue for the more deeply rooted coordination of the mechanisms underlying visual and linguistic processing, and for jointly considering the behavioral and neural correlates of scene-sentence reconciliation during situated comprehension. The Coordinated Interplay Account (CIA; Knoeferle & Crocker, 2007) asserts that incremental linguistic interpretation actively directs attention in the visual environment, thereby increasing the salience of attended scene information for comprehension. We review behavioral and neuroscientific findings in support of the CIA's three processing stages: (i) incremental sentence interpretation, (ii) language-mediated visual attention, and (iii) the on-line influence of non-linguistic visual context. We then describe a recently developed connectionist model which both embodies the central CIA proposals and has been successfully applied in modeling a range of behavioral findings from the visual world paradigm (Mayberry, Crocker, & Knoeferle, in press). Results from a new simulation suggest the model also correlates with event-related brain potentials elicited by the immediate use of visual context for linguistic disambiguation (Knoeferle, Habets, Crocker, & Münte, 2008). Finally, we argue that the mechanisms underlying interpretation, visual attention, and scene apprehension are not only in close temporal synchronization, but have co-adapted to optimize real-time visual grounding of situated spoken language, thus facilitating the association of linguistic, visual and motor representations that emerge during the course of our embodied linguistic experience in the world.

A Neurocomputational Model of the N400 and the P600 in Language Processing

“…Crocker et al., 2010; Mayberry et al., 2009). These thematic‐role assignment representations are 300‐dimensional vectors, which are divided into three 100‐dimensional slots.…”

Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

“…Cambridge: Cambridge University Press. Mayberry, 2010). The model's task is to predict thematic roles fillers in a target output representation during sentence processing.…”

“…However, these accounts, just like the above language-centric accounts, do not aim to model the nature of the interplay between comprehension processes, (visual attention) and information from the non-linguistic visual context (see Altmann & Mirković, 2009;Crocker, Knoeferle, & Mayberry, 2010;Knoeferle & Crocker, 2006Mayberry, Crocker, & Knoeferle, 2009).…”

Expectation-based Comprehension: Modeling the Interaction of World Knowledge and Linguistic Experience

“…Brouwer et al, (2012) adapt this perspective on the N400 as part of the Retrieval-Integration hypothesis and argue that the process of integrating word meaning with the unfolding utterance representation is instead reflected in P600 amplitude, a late positive deflection of the ERP signal. Under this view, our comprehension-centric formalization of surprisal is predicted to be reflected in the P600; surprisal is a measure of how likely a transition is from one interpretative state to the next, and P600 amplitude is a reflection of the neurophysiological processing involved in this transition (see also Brouwer, 2014;Brouwer, Crocker, Venhuizen, & Hoeks, 2017;Crocker, Knoeferle, & Mayberry, 2010).…”