Benchmarking n-grams, Topic Models and Recurrent Neural Networks by Cloze Completions, EEGs and Eye Movements

Hofmann, Markus; Biemann, Chris; Remus, Steffen

doi:10.1016/b978-1-78548-253-3.50010-x

Cited by 8 publications

(21 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When the semantic distances between questions and answers are used to account for such associative judgments, word2vec can nicely account for this conjunction fallacy. Hofmann, Biemann, and Remus (2017) also relied on model when accounting for human cloze completion probabilities, as well as event-related potentials and eye movement parameters previously accounted for by cloze completions. The CBOW model not only accounted for N400 effects, but also for single-fixation durations, that is, for the duration of a single fixation that was sufficient for successfully recognizing a word.…”

Section: Word2vec Cosinementioning

confidence: 99%

Simple Co‐Occurrence Statistics Reproducibly Predict Association Ratings

et al. 2018

Self Cite

View full text Add to dashboard Cite

What determines human ratings of association? We planned this paper as a test for association strength (AS) that is derived from the log likelihood that two words co-occur significantly more often together in sentences than is expected from their single word frequencies. We also investigated the moderately correlated interactions of word frequency, emotional valence, arousal, and imageability of both words (r's ≤ .3). In three studies, linear mixed effects models revealed that AS and valence reproducibly account for variance in the human ratings. To understand further correlated predictors, we conducted a hierarchical cluster analysis and examined the predictors of four clusters in competitive analyses: Only AS and word2vec skip-gram cosine distances reproducibly accounted for variance in all three studies. The other predictors of the first cluster (number of common associates, (positive) point-wise mutual information, and word2vec CBOW cosine) did not reproducibly explain further variance. The same was true for the second cluster (word frequency and arousal); the third cluster (emotional valence and imageability); and the fourth cluster (consisting of joint frequency only). Finally, we discuss emotional valence as an important dimension of semantic space. Our results suggest that a simple definition of syntagmatic word contiguity (AS) and a paradigmatic measure of semantic similarity (skip-gram cosine) provide the most general performance-independent explanation of association ratings.

show abstract

Section: Word2vec Cosinementioning

confidence: 99%

Simple Co‐Occurrence Statistics Reproducibly Predict Association Ratings

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…The second section is dedicated to latent semantic dimensions (Griffiths et al, 2007;Landauer & Dumais, 1997) that determine the co-occurrence of words in documents. Therefore, these models consolidate relatively long-range semantic relations (Hofmann, Biemann, & Remus, 2017), in contrast to short-range (preceding sentence context) consolidation dominant in the other two subsections.…”

Section: Language Models In Eye Movement Researchmentioning

confidence: 92%

“…Andrews, Vigliocco & Vinson, 2009), to our knowledge no other study so far addressed a topic model directly predicting reading times (but cf. Hofmann et al, 2017).…”

Section: Latent Semantic Dimensionsmentioning

confidence: 99%

“…While the topic models made only a minor contribution, the RNNs performed nearly as well as the n-gram models. In a second step, Hofmann et al (2017) predicted SFD data by the three LMs and compared them to CCP. In this case, the RNNs performed slightly better than the n-gram models, while the topics models made a notable, but lower contribution to SFD data.…”

Section: The Present Studymentioning

confidence: 99%

See 1 more Smart Citation

Language models explain word reading times better than empirical predictability

Mj¹,

Remus²,

Biemann³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

While word predictability from sentence context is typically investigated by cloze completion probabilities (CCP), it can be more deeply understood by relying on language models (LMs), allowing to define the three key components of memory: Memory starts with experience as implemented by a text corpus, here defined by Wikipedia capturing general knowledge and (movie) subtitles approximating social interactions. LMs then consolidate a long-term memory structure from experience, as addressed by n-gram, topics and recurrent neural network (RNN) models. Retrieval was investigated by predicting fixation durations from an English and a German reading sample. Item-level regressions showed greater correlations of LMs with single-fixation duration (SFD), gaze duration (GD) and total viewing time (TVT) than CCP. When predicting each fixation case separately using generalized additive models, three LMs together always performed better than CCP. When testing single LMs against the typically-sized English CCP sample (N = 30), LMs usually performed better than CCP (8 vs. 3). The larger German CCP sample (N = 272), however, often performed better than single LMs (4 vs. 2). Subtitles-trained n-gram probabilities of present (and last) words allowed for reliable predictions of all fixation durations. Wikipedia-trained topic probabilities of the last and present word allow for reliable predictions of late GD and TVT effects. The present word predictions of RNNs were less sensitive to training-corpus choice and are recommendable if a single LM is used. Moreover, its reliable next word probability effects make it most suitable to address parafoveal preview and top-down predictions.

show abstract

“…Frank (2009) found that an RNN provides significantly larger correlations with GD compared to a surprisal measure of the grammatical category of the word. An RNN, a topics model and a 5-gram model together explained not only about half of the variance of CCP, but they also performed significantly better than CCP in predicting SFD data (Hofmann, Biemann, & Remus, 2017).…”

Section: Introductionmentioning

confidence: 97%

Decomposing predictability: Semantic feature overlap between words and the dynamics of reading for meaning

A.²,

Roelke³

et al. 2019

Preprint

View full text Add to dashboard Cite

The present study uses a computational approach to examine the role of semantic constraints in normal reading. This methodology avoids confounds inherent in conventional measures of predictability, allowing for theoretically deeper accounts of semantic processing. We start from a definition of associations between words based on the significant log likelihood that two words co-occur frequently together in the sentences of a large text corpus. Direct associations between stimulus words were controlled, and semantic feature overlap between prime and target words was manipulated by their common associates. The stimuli consisted of sentences of the form pronoun, verb, article, adjective and noun, followed by a series of closed class words, e. g. "She rides the grey elephant on one of her many exploratory voyages". The results showed that verbnoun overlap reduces single and first fixation durations of the target noun and adjective-noun overlap reduces go-past durations. A dynamic spreading of activation account suggests that associates of the prime words take some time to become activated: The verb can act on the target noun's early eye-movement measures presented three words later, while the adjective is presented immediately prior to the target, which induces sentence re-examination after a difficult adjectivenoun semantic integration.

show abstract

Benchmarking n-grams, Topic Models and Recurrent Neural Networks by Cloze Completions, EEGs and Eye Movements

Cited by 8 publications

References 30 publications

Simple Co‐Occurrence Statistics Reproducibly Predict Association Ratings

Simple Co‐Occurrence Statistics Reproducibly Predict Association Ratings

Language models explain word reading times better than empirical predictability

Decomposing predictability: Semantic feature overlap between words and the dynamics of reading for meaning

Contact Info

Product

Resources

About