Learning to Attend: A Connectionist Model of Situated Language Comprehension

Mayberry, Marshall R.; Crocker, Matthew W.; Knoeferle, Pia

doi:10.1111/j.1551-6709.2009.01019.x

Cited by 53 publications

(69 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Few attempts have been made to model the interaction of linguistic and visual processing in a single model, which would then be able to account for visual world data such as the one resported in the present article. A notable exception is the Coordinated Interplay Account (CIA) proposed by Knoeferle and Crocker (2007) and Mayberry, Knoeferle, and Crocker (2009). The CIA model is implemented as an SRN, which in addition to textual input also takes a representation of the events in a visual scene as input.…”

Section: Discussionmentioning

confidence: 99%

The use of verb-specific information for prediction in sentence processing

Arai

Keller

2013

Language and Cognitive Processes

View full text Add to dashboard Cite

Recent research has shown that language comprehenders make predictions about upcoming linguistic information. These studies demonstrate that the processor not only analyses the input that it received but also predicts upcoming unseen elements. Two visual world experiments were conducted to examine the type of syntactic information this prediction process has access to.Experiment 1 examined whether the verb's subcategorization information is used for predicting a direct object, by comparing transitive verbs (e.g., punish) to intransitive verbs (e.g., disagree).Experiment 2 examined whether verb frequency information is used for predicting a reduced relative clause by contrasting verbs that are infrequent in the past participle form (e.g., watch) with ones that are frequent in that form (e.g., record). Both experiments showed that comprehenders used lexically specific syntactic information to predict upcoming syntactic structure; this information can be used to avoid garden paths in certain cases, as Experiment 2 demonstrated.

show abstract

Section: Discussionmentioning

confidence: 99%

The use of verb-specific information for prediction in sentence processing

Arai

Keller

2013

Language and Cognitive Processes

View full text Add to dashboard Cite

show abstract

“…However, these accounts, just like the above language-centric accounts, do not aim to model the nature of the interplay between comprehension processes, (visual attention) and information from the non-linguistic visual context (see Altmann & Mirković, 2009;Crocker, Knoeferle, & Mayberry, 2010;Knoeferle & Crocker, 2006Mayberry, Crocker, & Knoeferle, 2009).…”

Section: Visually Situated Language Comprehension: Methodological Advmentioning

confidence: 99%

Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

Knoeferle

2015

Cognitive Neuroscience of Natural Language Use

Self Cite

View full text Add to dashboard Cite

The present chapter reviews the literature on visually situated language comprehension against the background that most theories of real-time sentence comprehension have ignored rich non-linguistic contexts. However, listeners' eye movements to objects during spoken language comprehension, as well as their eventrelated brain potentials (ERPs) have revealed that non-linguistic cues play an important role for real-time comprehension. In fact, referential processes are rapid and central in visually situated spoken language comprehension and even abstract words are rapidly grounded in objects through semantic associations. Similar ERP responses for non-linguistic and linguistic effects on comprehension suggest these two information sources are on a par in informing language comprehension. ERPs further revealed that non-linguistic cues affect lexical-semantic as well as compositional processes, thus further cementing the role of rich non-linguistic context in language comprehension. However, there is also considerable ambiguity in the linking between comprehension processes and each of these two measures (eye movements and ERPs). Combining eye-tracking and event-related brain potentials would improve the interpretation of individual measures and thus insights into visually-situated language comprehension.Knoeferle, P. (to appear). Language comprehension in rich non-linguistic contexts: combining eye tracking and event-related brain potentials. In: Roel, Williams (Ed.). Towards a cognitive neuroscience of natural language use. Cambridge: Cambridge University Press.

show abstract

“…These models are often given a connectionist implementation rooted in the notion of activation (Spivey & Tanenhaus, 1998;Tabor & Tanenhaus, 1999;Christiansen & Chater, 1999;Rohde, 2002;Mayberry, Crocker, & Knoeferle, 2009). Besides lexicalism, these models differ from the "symbolist" account advanced in this paper in virtue of being based upon an activation-based associative memory (Kohonen, 1989;McElree, Foraker, & Dyer, 2003).…”

Section: Future Workmentioning

confidence: 99%

Parallel processing and sentence comprehension difficulty

Boston

Hale

Vasishth

et al. 2011

Language and Cognitive Processes

119

129

View full text Add to dashboard Cite

Eye fixation durations during normal reading correlate with processing difficulty but the specific cognitive mechanisms reflected in these measures are not well understood. This study finds support in German readers' eye fixations for two distinct difficulty metrics: surprisal, which reflects the change in probabilities across syntactic analyses as new words are integrated, and retrieval, which quantifies comprehension difficulty in terms of working memory constraints. We examine the predictions of both metrics using a family of dependency parsers indexed by an upper limit on the number of candidate syntactic analyses they retain at successive words. Surprisal models all fixation measures and regression probability. By contrast, retrieval does not model any measure in serial processing. As more candidate analyses are considered in parallel at each word, retrieval can account for the same measures as surprisal. This pattern suggests an important role for ranked parallelism in theories of sentence comprehension.

show abstract

Learning to Attend: A Connectionist Model of Situated Language Comprehension

Cited by 53 publications

References 50 publications

The use of verb-specific information for prediction in sentence processing

The use of verb-specific information for prediction in sentence processing

Language comprehension in rich non-linguistic contexts: combining eye-tracking and event-related brain potentials

Parallel processing and sentence comprehension difficulty

Contact Info

Product

Resources

About