CxGBERT: BERT meets Construction Grammar

Madabushi, Harish Tayyar; Romain, Laurence; Divjak, Dagmar; Milin, Petar

doi:10.18653/v1/2020.coling-main.355

Cited by 13 publications

(9 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…So far, relatively few papers approached LM probing from a construction grammar perspective. Madabushi et al (2020) probed for BERT's knowledge of constructions via a sentence pair classification task of predicting whether two sentences share the same construction. Their probe was based on data from Dunn (2017), who used an unsupervised algorithm to extract plausible constructions from corpora based on association strength.…”

Section: Linguistic Probing Of Lmsmentioning

confidence: 99%

Neural reality of argument structure constructions

Li¹,

Zhu²,

Thomas³

et al. 2022

Preprint

View full text Add to dashboard Cite

In lexicalist linguistic theories, argument structure is assumed to be predictable from the meaning of verbs. As a result, the verb is the primary determinant of the meaning of a clause. In contrast, construction grammarians propose that argument structure is encoded in constructions (or form-meaning pairs) that are distinct from verbs. Decades of psycholinguistic research have produced substantial empirical evidence in favor of the construction view. Here we adapt several psycholinguistic studies to probe for the existence of argument structure constructions (ASCs) in Transformerbased language models (LMs). First, using a sentence sorting experiment, we find that sentences sharing the same construction are closer in embedding space than sentences sharing the same verb. Furthermore, LMs increasingly prefer grouping by construction with more input data, mirroring the behaviour of non-native language learners. Second, in a "Jabberwocky" priming-based experiment, we find that LMs associate ASCs with meaning, even in semantically nonsensical sentences. Our work offers the first evidence for ASCs in LMs and highlights the potential to devise novel probing methods grounded in psycholinguistic research.

show abstract

Section: Linguistic Probing Of Lmsmentioning

confidence: 99%

Neural reality of argument structure constructions

Li¹,

Zhu²,

Thomas³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other studies have approached the question by modifying BERT's training scheme. Tayyar Madabushi et al (2020) train a BERT variant where the next sentence prediction task has been replaced with a same construction prediction task and find mixed results on downstream tasks. Levine et al (2020) train a BERT variant by adding a new supersense prediction task, wherein the masked token's WordNet supersense is to be predicted, and find performance gains on a variety of meaning-related tasks, which shows that BERT's representations do not perfectly capture word senses.…”

Section: Word Sense Bertologymentioning

confidence: 99%

BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTology

Gessler¹,

Schneider²

2021

Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

View full text Add to dashboard Cite

An important question concerning contextualized word embedding (CWE) models like BERT is how well they can represent different word senses, especially those in the long tail of uncommon senses. Rather than build a WSD system as in previous work, we investigate contextualized embedding neighborhoods directly, formulating a query-by-example nearest neighbor retrieval task and examining ranking performance for words and senses in different frequency bands. In an evaluation on two English sense-annotated corpora, we find that several popular CWE models all outperform a random baseline even for proportionally rare senses, without explicit sense supervision. However, performance varies considerably even among models with similar architectures and pretraining regimes, with especially large differences for rare word senses, revealing that CWE models are not all created equal when it comes to approximating word senses in their native representations.

show abstract

“…A constructional approach to language focuses on symbolic formmeaning mappings that are potentially idiomatic. Previous work on computational CxG has explored how to discover potential constructions (Wible and Tsao, 2010;Forsberg et al, 2014;Dunn, 2017), the process of construction learning , and whether constructional information is implicitly encoded in models like BERT (Tayyar Madabushi et al, 2020).…”

Section: Exposure and Convergencementioning

confidence: 99%

Learned Construction Grammars Converge Across Registers Given Increased Exposure

Dunn

Madabushi

2021

Proceedings of the 25th Conference on Computational Natural Language Learning

Self Cite

View full text Add to dashboard Cite

This paper measures the impact of increased exposure on whether learned construction grammars converge onto shared representations when trained on data from different registers. Register influences the frequency of constructions, with some structures common in formal but not informal usage. We expect that a grammar induction algorithm exposed to different registers will acquire different constructions. To what degree does increased exposure lead to the convergence of register-specific grammars? The experiments in this paper simulate language learning in 12 languages (half Germanic and half Romance) with corpora representing three registers (Twitter, Wikipedia, Web). These simulations are repeated with increasing amounts of exposure, from 100k to 2 million words, to measure the impact of exposure on the convergence of grammars. The results show that increased exposure does lead to converging grammars across all languages. In addition, a shared core of register-universal constructions remains constant across increasing amounts of exposure.

show abstract

CxGBERT: BERT meets Construction Grammar

Cited by 13 publications

References 79 publications

Neural reality of argument structure constructions

Neural reality of argument structure constructions

BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTology

Learned Construction Grammars Converge Across Registers Given Increased Exposure

Contact Info

Product

Resources

About