What’s in a Name? Are BERT Named Entity Representations just as Good for any other Name?

Balasubramanian, Sriram; Jain, Naman; Jindal, Gaurav; Awasthi, Abhijeet; Sarawagi, Sunita

doi:10.18653/v1/2020.repl4nlp-1.24

Cited by 17 publications

(13 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Out-of-the-box BERT is surprisingly brittle to named entity replacements: For example, replacing names in the coreference task changes 85% of predictions (Balasubramanian et al, 2020). This suggests that the model does not actually form a generic idea of named entities, although its F1 scores on NER probing tasks are high (Tenney et al, 2019a).…”

Section: Semantic Knowledgementioning

confidence: 99%

A Primer in BERTology: What We Know About How BERT Works

Rogers

Kovaleva

Rumshisky

2020

Transactions of the Association for Computational Linguistics

957

590

View full text Add to dashboard Cite

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.

show abstract

Section: Semantic Knowledgementioning

confidence: 99%

A Primer in BERTology: What We Know About How BERT Works

Rogers

Kovaleva

Rumshisky

2020

Transactions of the Association for Computational Linguistics

957

590

View full text Add to dashboard Cite

show abstract

“…Recent works have noted that over-relying on entity name information negatively impacts NLU tasks. Balasubramanian et al (2020) found that substituting named-entities in standard test sets of natural language inference, coreference resolution, and grammar error correction has a negative impact on those tasks. In political claims detection (Padó et al, 2019), Dayanik and Padó (2020) show that claims made by frequently occurring politicians in the training data are better recognized than those made by less frequent ones.…”

Section: Diagnosing Biaismentioning

confidence: 99%

Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition

Ghaddar

Langlais

Ahmad

et al. 2021

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

In this work, we examine the ability of NER models to use contextual information when predicting the type of an ambiguous entity. We introduce NRB, a new testbed carefully designed to diagnose Name Regularity Bias of NER models. Our results indicate that all state-of-the-art models we tested show such a bias; BERT fine-tuned models significantly outperforming feature-based (LSTM-CRF) ones on NRB, despite having comparable (sometimes lower) performance on standard benchmarks. To mitigate this bias, we propose a novel model-agnostic training method that adds learnable adversarial noise to some entity mentions, thus enforcing models to focus more strongly on the contextual signal, leading to significant gains on NRB. Combining it with two other training strategies, data augmentation and parameter freezing, leads to further gains.

show abstract

“…As an example, BLESS (Baroni and Lenci, 2011) includes goldstandard annotations for only 200 concepts. Other approaches have tested semantic ability by using prompt-engineering and inspecting the predictions of the models (Petroni et al, 2019;Ettinger, 2020;Talmor et al, 2020), but other works have also shown a high variability in the results depending on the prompt design (Balasubramanian et al, 2020;Reynolds and McDonell, 2021;Zhao et al, 2021).…”

Section: Automated Extraction Of Concept Relationsmentioning

confidence: 99%

Inspecting the concept knowledge graph encoded by modern language models

Aspillaga¹,

Mendoza²,

Soto³

2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

The field of natural language understanding has experienced exponential progress in the last few years, with impressive results in several tasks. This success has motivated researchers to study the underlying knowledge encoded by these models. Despite this, attempts to understand their semantic capabilities have not been successful, often leading to non-conclusive, or contradictory conclusions among different works. Via a probing classifier, we extract the underlying knowledge graph of nine of the most influential language models of the last years, including word embeddings, text generators, and context encoders. This probe is based on concept relatedness, grounded on WordNet. Our results reveal that all the models encode this knowledge, but suffer from several inaccuracies. Furthermore, we show that the different architectures and training strategies lead to different model biases. We conduct a systematic evaluation to discover specific factors that explain why some concepts are challenging. We hope our insights will motivate the development of models that capture concepts more precisely.

show abstract

What’s in a Name? Are BERT Named Entity Representations just as Good for any other Name?

Cited by 17 publications

References 25 publications

A Primer in BERTology: What We Know About How BERT Works

A Primer in BERTology: What We Know About How BERT Works

Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognition

Inspecting the concept knowledge graph encoded by modern language models

Contact Info

Product

Resources

About