Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

Shwartz, Vered; Dagan, Ido

doi:10.1162/tacl_a_00277

Cited by 53 publications

(65 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The BERT model has been a particular focus of analysis work since its introduction. Previous work has focused on analyzing the attention mechanism (Vig and Belinkov, 2019;Clark et al, 2019), parameters Radford et al, 2019; and embeddings (Shwartz and Dagan, 2019;Liu et al, 2019a). We build on this work with a particular, controlled focus on the evolution of phrasal representation in a variety of state-of-the-art transformers.…”

Section: Related Workmentioning

confidence: 99%

Assessing Phrasal Representation and Composition in Transformers

Ettinger

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Deep transformer models have pushed performance on NLP tasks to new limits, suggesting sophisticated treatment of complex linguistic inputs, such as phrases. However, we have limited understanding of how these models handle representation of phrases, and whether this reflects sophisticated composition of phrase meaning like that done by humans. In this paper, we present systematic analysis of phrasal representations in state-of-the-art pre-trained transformers. We use tests leveraging human judgments of phrase similarity and meaning shift, and compare results before and after control of word overlap, to tease apart lexical effects versus composition effects. We find that phrase representation in these models relies heavily on word content, with little evidence of nuanced composition. We also identify variations in phrase representation quality across models, layers, and representation types, and make corresponding recommendations for usage of representations from these models.

show abstract

Section: Related Workmentioning

confidence: 99%

Assessing Phrasal Representation and Composition in Transformers

Ettinger

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…Recently, Shwartz and Dagan (2019) found that while these representations excel at detecting non-compositional noun compounds, they perform much worse at revealing implicit information such as the relationship between the constituents. Moreover, looking into these models' predictions of substitute constituents shows that even when they recognize a constituent is not used in its literal sense (e.g.…”

Section: Discussionmentioning

confidence: 99%

A Systematic Comparison of English Noun Compound Representations

Shwartz¹

2019

Proceedings of the Joint Workshop on Multiword Expressions and WordNet (MWE-WN 2019)

Self Cite

View full text Add to dashboard Cite

Building meaningful representations of noun compounds is not trivial since many of them scarcely appear in the corpus. To that end, composition functions approximate the distributional representation of a noun compound by combining its constituent distributional vectors. In the more general case, phrase embeddings have been trained by minimizing the distance between the vectors representing paraphrases. We compare various types of noun compound representations, including distributional, compositional, and paraphrasebased representations, through a series of tasks and analyses, and with an extensive number of underlying word embeddings. We find that indeed, in most cases, composition functions produce higher quality representations than distributional ones, and they improve with computational power. No single function performs best in all scenarios, suggesting that a joint training objective may produce improved representations.

show abstract

“…In particular, the representations produced by BERT (Devlin et al, 2019) have been used to create high performing models for many language understanding tasks, although their status as a linguistically sound model of meaning is debated (Mickus et al, 2020). Shwartz and Dagan (2019) test BERT on several cases of figurative language, but to the best of our knowledge BERT ability to identify metonymy has never been addressed yet.…”

Section: Related Workmentioning

confidence: 99%

Don’t Invite BERT to Drink a Bottle: Modeling the Interpretation of Metonymies Using BERT and Distributional Representations

Pedinotti

Lenci

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

In this work, we carry out two experiments in order to assess the ability of BERT to capture the meaning shift associated with metonymic expressions. We test the model on a new dataset that is representative of the most common types of metonymy. We compare BERT with the Structured Distributional Model (SDM), a model for the representation of words in context which is based on the notion of Generalized Event Knowledge. The results reveal that, while BERT ability to deal with metonymy is quite limited, SDM is good at predicting the meaning of metonymic expressions, providing support for an account of metonymy based on event knowledge.

show abstract

Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

Cited by 53 publications

References 36 publications

Assessing Phrasal Representation and Composition in Transformers

Assessing Phrasal Representation and Composition in Transformers

A Systematic Comparison of English Noun Compound Representations

Don’t Invite BERT to Drink a Bottle: Modeling the Interpretation of Metonymies Using BERT and Distributional Representations

Contact Info

Product

Resources

About