Reading behavior predicts syntactic categories

Barrett, Maria; Søgaard, Anders

doi:10.18653/v1/k15-1038

Cited by 33 publications

(31 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…I convert the fMRI data and the eye-tracking data to vectors of aggregate statistics following the suggestions in Barrett and Søgaard (2015) and Barrett et al (2016). Table 1 presents the nearest neighbors (out of the 9 randomly selected words) in the gold data, as well as the two word embeddings.…”

Section: Sprachspiel Theorymentioning

confidence: 99%

Evaluating word embeddings with fMRI and eye-tracking

Søgaard¹

2016

Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP

Self Cite

View full text Add to dashboard Cite

The workshop CfP assumes that downstream evaluation of word embeddings is impractical, and that a valid evaluation metric for pairs of word embeddings can be found. I argue below that if so, the only meaningful evaluation procedure is comparison with measures of human word processing in the wild. Such evaluation is non-trivial, but I present a practical procedure here, evaluating word embeddings as features in a multi-dimensional regression model predicting brain imaging or eyetracking word-level aggregate statistics.What's the meaning of embeddings? In order to decide how to evaluate word embeddings, we first need to decide what word embeddings are supposed to encode. If we assume that word embeddings are primarily representations of the meaning of words, it makes sense to consult lexical semantic theories.Here's a very, very, very (very, . . . ) crude characterization of lexical semantics: Researchers disagree whether words are defined by their cooccurrences (Firth, 1957), the contexts in which they are used (Wittgenstein, 1953), how they are organized in the brain (Miller and Fellbaum, 1992), or the referents they denote in the real world (Montague, 1973). I realize this is a ridiculously simplistic reduction of modern lexical semantics, but I think it suffices for our discussion of how best to evaluate word embeddings. 1Any metrics here? From (one or more of) these theories we want to derive a valid evaluation metric. In my view, a valid metric satisfies two principles: (i) that it measures what we want to measure (adequacy), and (ii) that it cannot easily be 1 See the discussion in the last paragraph.hacked. What I mean by (i) is that we want word embeddings to capture the meaning of words; and by (ii), that the reason we want to play the evaluation game is because it isn't obvious what the meaning of a word is. If the meaning of a word was given directly by its character sequence, I would not be writing this paper, and this workshop would not have been proposed. The question then is, do any of the four theories above provide us with a valid metric for the general quality of word embeddings?Below, I argue that none of the four theories leave us with fully valid evaluation metrics, except maybe COGNITIVE LEXICAL SEMANTICS. I suggest evaluating embeddings by direct comparison with brain-imaging and eye-tracking data rather than word association norms, as an alternative approach to COGNITIVE LEXICAL SEMAN-TICS. I show that state-of-the-art embeddings correlate poorly with such data, but argue that this is nevertheless the only valid metric left on the table, if downstream evaluation is not an option -and that, practically, we can evaluate embeddings by the error of a multi-dimensional regression model predicting brain imaging or eye-tracking data using the embeddings as features.Co-occurrence theory In CO-OCCURRENCE THEORY, the meaning of a word is defined by its co-occurrences with other words -e.g., the meaning of big is given by its co-occurrence with words such as house and small, i.e., its value i...

show abstract

Section: Sprachspiel Theorymentioning

confidence: 99%

Evaluating word embeddings with fMRI and eye-tracking

Søgaard¹

2016

Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP

Self Cite

View full text Add to dashboard Cite

show abstract

“…The data comes from (Barrett and Søgaard, 2015) and is publicly available 1 . In this experiment 10 native English speakers read 250 syntactically annotated sentences in English (min.…”

Section: Eye Tracking Datamentioning

confidence: 99%

“…Recently, Barrett and Søgaard (2015) presented evidence that gaze features can be used to discriminate between most pairs of parts of speech (POS). Their study uses all the coarse-grained POS labels proposed by Petrov et al (2011).…”

Section: Introductionmentioning

confidence: 99%

Using reading behavior to predict grammatical functions

Barrett

Søgaard

2015

Proceedings of the Sixth Workshop on Cognitive Aspects of Computational Language Learning

Self Cite

View full text Add to dashboard Cite

This paper investigates to what extent grammatical functions of a word can be predicted from gaze features obtained using eye-tracking. A recent study showed that reading behavior can be used to predict coarse-grained part of speech, but we go beyond this, and show that gaze features can also be used to make more finegrained distinctions between grammatical functions, e.g., subjects and objects. In addition, we show that gaze features can be used to improve a discriminative transition-based dependency parser.

show abstract

“…Eye trackers and gaze features collected from them have been recently applied to natural language processing (NLP) tasks, such as part-ofspeech tagging (Duffy et al, 1988;Nilsson and Nivre, 2009;Barrett and Søgaard, 2015a), namedentity recognition (Tokunaga et al, 2017) or readability (González-Garduño and Søgaard, 2018). Eye-movement data has been also used for parsing.…”

Section: Introductionmentioning

confidence: 99%

“…However, even if in the near future every user has an eye tracker on top of their screen -a scenario which is far from guaranteed, and raises privacy concerns (Liebling and Preibusch, 2014)many running NLP applications that process data from various Internet sources will not expect to have any human being reading massive amounts of data. Other studies (Barrett and Søgaard, 2015a; instead derive gaze features from the training set: in particular, they collect type-level gaze features from the vocabulary in the training set; which are then aggregated to create a lookup table and used as a sort of precomputed gaze input when a given word in the test set matches an entry, otherwise, a token has an unknown gaze feature value. In this manner, the influence of gaze features on unseen data depends on the vocabulary encountered during training.…”

Section: Introductionmentioning

confidence: 99%

Towards Making a Dependency Parser See

Strzyz¹,

Vilares²,

Gómez-Rodríguez³

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

We explore whether it is possible to leverage eye-tracking data in an RNN dependency parser (for English) when such information is only available during training -i.e. no aggregated or token-level gaze features are used at inference time. To do so, we train a multitask learning model that parses sentences as sequence labeling and leverages gaze features as auxiliary tasks. Our method also learns to train from disjoint datasets, i.e. it can be used to test whether already collected gaze features are useful to improve the performance on new non-gazed annotated treebanks. Accuracy gains are modest but positive, showing the feasibility of the approach. It can serve as a first step towards architectures that can better leverage eye-tracking data or other complementary information available only for training sentences, possibly leading to improvements in syntactic parsing.

show abstract

Reading behavior predicts syntactic categories

Cited by 33 publications

References 8 publications

Evaluating word embeddings with fMRI and eye-tracking

Evaluating word embeddings with fMRI and eye-tracking

Using reading behavior to predict grammatical functions

Towards Making a Dependency Parser See

Contact Info

Product

Resources

About