The influence of recent scene events on spoken comprehension: Evidence from eye movements

Knoeferle, Pia; Crocker, Matthew W.

doi:10.1016/j.jml.2007.01.003

Cited by 106 publications

(141 citation statements)

References 17 publications

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…Our study suggests that language production is influenced by a similar set of factors (structural, conceptual, and perceptual). However, the direction of multi-modal integration in production is reversed, i.e., scene understanding has to happen before constraints supplied by visual attention are applied to sentence production (Altmann & Kamide, 1999;Knoeferle & Crocker, 2007;Kukona, Fang, Aicher, Chen, & Magnuson, 2011). Future research is needed to investigate the similarities and differences between situated comprehension and production in more detail, perhaps in a dialogue setting in which both processes happen at the same time.…”

Section: Discussionmentioning

confidence: 99%

Integrating mechanisms of visual guidance in naturalistic language production

Coco

Keller

2014

Cogn Process

View full text Add to dashboard Cite

Situated language production requires the integration of visual attention and linguistic processing. Previous work has not conclusively disentangled the role of perceptual scene information and structural sentence information in guiding visual attention. In this paper, we present an eye-tracking study that demonstrates that three types of guidance, perceptual, conceptual, and structural, interact to control visual attention. In a cued language production experiment, we manipulate perceptual (scene clutter) and conceptual guidance (cue animacy), and measure structural guidance (syntactic complexity of the utterance). Analysis of the time course of language production, before and during speech, reveals that all three forms of guidance affect the complexity of visual responses, quantified in terms of the entropy of attentional landscapes and the turbulence of scan patterns, especially during speech. We find that perceptual and conceptual guidance mediate the distribution of attention in the scene, whereas structural guidance closely relates to scan-pattern complexity. Furthermore, the eye-voice span of the cued object and its perceptual competitor are similar; its latency mediated by both perceptual and structural guidance. These results rule out a strict interpretation of structural guidance as the single dominant form of visual guidance in situated language production. Rather, the phase of the task and the associated demands of cross-modal cognitive processing determine the mechanisms that guide attention.

show abstract

Section: Discussionmentioning

confidence: 99%

Integrating mechanisms of visual guidance in naturalistic language production

Coco

Keller

2014

Cogn Process

View full text Add to dashboard Cite

show abstract

“…A similar preference emerged when the choice was between the target of a recently acted-upon object and another target of a future action. In this situation, listeners rapidly inspected the target of the recent action (e.g., a candelabra that had been polished) in preference to the target of a future polishing action (e.g., polishing crystal glasses, target-condition assignment was counterbalanced, Knoeferle & Crocker, 2007). The recent-event preference replicated with real-world events (Knoeferle, Carminati, Abashidze, & Essig, 2011a) and when the within-experiment frequency of future (relative to recent) actions was increased to 75 (vs. 25) percent (Abashidze, Knoeferle, & Carminati, 2013).…”

Section: Visually Situated Language Comprehension: Methodsological Advmentioning

confidence: 99%

Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

Knoeferle

2015

Cognitive Neuroscience of Natural Language Use

Self Cite

View full text Add to dashboard Cite

The present chapter reviews the literature on visually situated language comprehension against the background that most theories of real-time sentence comprehension have ignored rich non-linguistic contexts. However, listeners' eye movements to objects during spoken language comprehension, as well as their eventrelated brain potentials (ERPs) have revealed that non-linguistic cues play an important role for real-time comprehension. In fact, referential processes are rapid and central in visually situated spoken language comprehension and even abstract words are rapidly grounded in objects through semantic associations. Similar ERP responses for non-linguistic and linguistic effects on comprehension suggest these two information sources are on a par in informing language comprehension. ERPs further revealed that non-linguistic cues affect lexical-semantic as well as compositional processes, thus further cementing the role of rich non-linguistic context in language comprehension. However, there is also considerable ambiguity in the linking between comprehension processes and each of these two measures (eye movements and ERPs). Combining eye-tracking and event-related brain potentials would improve the interpretation of individual measures and thus insights into visually-situated language comprehension.Knoeferle, P. (to appear). Language comprehension in rich non-linguistic contexts: combining eye tracking and event-related brain potentials. In: Roel, Williams (Ed.). Towards a cognitive neuroscience of natural language use. Cambridge: Cambridge University Press.

show abstract

“…This is accomplished by three temporally dependent processing stages: incremental sentence interpretation (step i), utterance mediated attention (step i ) towards mentioned or anticipated scene entities, and finally scene integration which reconciles the interpretation with relevant scene information (step i ) (Knoeferle & Crocker, 2007).…”

Section: Sentence Interpretationmentioning

confidence: 99%

“…Based on recent behavioral and neuroscientific findings, however, we argue for the more deeply rooted coordination of the mechanisms underlying visual and linguistic processing, and for jointly considering the behavioral and neural correlates of scene-sentence reconciliation during situated comprehension. The Coordinated Interplay Account (CIA; Knoeferle & Crocker, 2007) asserts that incremental linguistic interpretation actively directs attention in the visual environment, thereby increasing the salience of attended scene information for comprehension. We review behavioral and neuroscientific findings in support of the CIA's three processing stages: (i) incremental sentence interpretation, (ii) language-mediated visual attention, and (iii) the on-line influence of non-linguistic visual context.…”

Section: Introductionmentioning

confidence: 99%

Situated sentence processing: The coordinated interplay account and a neurobehavioral model

2010

Self Cite

View full text Add to dashboard Cite

Empirical evidence demonstrating that sentence meaning is rapidly reconciled with the visual environment has been broadly construed as supporting the seamless interaction of visual and linguistic representations during situated comprehension. Based on recent behavioral and neuroscientific findings, however, we argue for the more deeply rooted coordination of the mechanisms underlying visual and linguistic processing, and for jointly considering the behavioral and neural correlates of scene-sentence reconciliation during situated comprehension. The Coordinated Interplay Account (CIA; Knoeferle & Crocker, 2007) asserts that incremental linguistic interpretation actively directs attention in the visual environment, thereby increasing the salience of attended scene information for comprehension. We review behavioral and neuroscientific findings in support of the CIA's three processing stages: (i) incremental sentence interpretation, (ii) language-mediated visual attention, and (iii) the on-line influence of non-linguistic visual context. We then describe a recently developed connectionist model which both embodies the central CIA proposals and has been successfully applied in modeling a range of behavioral findings from the visual world paradigm (Mayberry, Crocker, & Knoeferle, in press). Results from a new simulation suggest the model also correlates with event-related brain potentials elicited by the immediate use of visual context for linguistic disambiguation (Knoeferle, Habets, Crocker, & Münte, 2008). Finally, we argue that the mechanisms underlying interpretation, visual attention, and scene apprehension are not only in close temporal synchronization, but have co-adapted to optimize real-time visual grounding of situated spoken language, thus facilitating the association of linguistic, visual and motor representations that emerge during the course of our embodied linguistic experience in the world.

show abstract

The influence of recent scene events on spoken comprehension: Evidence from eye movements

Cited by 106 publications

References 17 publications

Integrating mechanisms of visual guidance in naturalistic language production

Integrating mechanisms of visual guidance in naturalistic language production

Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

Situated sentence processing: The coordinated interplay account and a neurobehavioral model

Contact Info

Product

Resources

About