Don’t Invite BERT to Drink a Bottle: Modeling the Interpretation of Metonymies Using BERT and Distributional Representations

Pedinotti, Paolo; Lenci, Alessandro

doi:10.18653/v1/2020.coling-main.602

Cited by 4 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various studies examine the Transformer-based language model BERT's (Devlin et al, 2019) ability to capture tropes like metonyms (Pedinotti and Lenci, 2020), idioms (Kurfalı and Östling, 2020), and multiple types of figurative language (Shwartz and Dagan, 2019). Kurfalı and Östling (2020) detect idioms based on the dissimilarity of BERT's representations of a PIE and its context, assuming that contextual discrepancies indicate figurative usage.…”

Section: Tropes In Transformermentioning

confidence: 99%

“…Kurfalı and Östling (2020) detect idioms based on the dissimilarity of BERT's representations of a PIE and its context, assuming that contextual discrepancies indicate figurative usage. Pedinotti and Lenci (2020) measure whether BERT detects meaning shift for metonymic expressions but find cloze probabilities more indicative than vector similarities. Shwartz and Dagan (2019) find that BERT is better at detecting figurative meaning shift than at predicting implicit meaning -e.g.…”

Section: Tropes In Transformermentioning

confidence: 99%

See 1 more Smart Citation

Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

Dankers¹,

Lucas²,

Titov³

2022

Preprint

View full text Add to dashboard Cite

Unlike literal expressions, idioms' meanings do not directly follow from their parts, posing a challenge for neural machine translation (NMT). NMT models are often unable to translate idioms accurately and over-generate compositional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language. When Transformer emits a non-literal translation -i.e. identifies the expression as idiomatic -the encoder processes idioms more strongly as single lexical units compared to literal expressions. This manifests in idioms' parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder's cross-attention, figurative inputs result in reduced attention on source-side tokens. These results suggest that Transformer's tendency to process idioms as compositional expressions contributes to literal translations of idioms.

show abstract

Section: Tropes In Transformermentioning

confidence: 99%

Section: Tropes In Transformermentioning

confidence: 99%

Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

Dankers¹,

Lucas²,

Titov³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…2020; Pedinotti and Lenci, 2020). As a consequence, they usually fall far behind the performance of the same PLM fine-tuned with (i) sense annotations (Hadiwinoto et al, 2019;Blevins and Zettlemoyer, 2020) or (ii) external (e.g., WordNet) knowledge (Levine et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

Liu¹,

Liu²,

Collier³

et al. 2021

Proceedings of the 25th Conference on Computational Natural Language Learning

View full text Add to dashboard Cite

Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques. Inspired by this line of work, in this paper we propose a fully unsupervised approach to improving word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MIRROR-WIC. The proposed method leverages only raw texts sampled from Wikipedia, assuming no sense-annotated data, and learns contextaware word representations within a standard contrastive learning setup. We experiment with a series of standard and comprehensive WiC benchmarks across multiple languages. Our proposed fully unsupervised MIRROR-WIC models obtain substantial gains over offthe-shelf PLMs across all monolingual, multilingual and cross-lingual setups. Moreover, on some standard WiC benchmarks, MIRROR-WIC is even on-par with supervised models fine-tuned with in-task data and sense labels.

show abstract

Evaluation of Arabic-Based Contextualized Word Embedding Models

Yagi

Mansour

Kamalov

et al. 2021

2021 International Conference on Asian Language Processing (IALP)

View full text Add to dashboard Cite

Don’t Invite BERT to Drink a Bottle: Modeling the Interpretation of Metonymies Using BERT and Distributional Representations

Cited by 4 publications

References 12 publications

Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

Evaluation of Arabic-Based Contextualized Word Embedding Models

Contact Info

Product

Resources

About