A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Williams, Adina; Nangia, Nikita; Bowman, Samuel R.

doi:10.48550/arxiv.1704.05426

Cited by 228 publications

(316 citation statements)

References 0 publications

Supporting

Mentioning

316

Contrasting

Order By: Relevance

“…More recent efforts focus on designing generic sentence-level learning objectives or tasks. On the supervised learning regime, Conneau et al (2017); Cer et al (2018) empirically show the effectiveness of leveraging the NLI task (Bowman et al, 2015a;Williams et al, 2017) to promote generic sentence representations. The task involves classifying each sentence pair into one of three categories: entailment, contradiction, or neutral.…”

Section: Related Workmentioning

confidence: 99%

“…We observe the same trend on the categorical-level evaluation, i.e., large inter class distances come along with large intra cluster distances. In contrast, the original pre-trained language models consistently show poor performance on all three down- 7 We take 10000 randomly sampled entailment pairs from the combination of SNLI (Bowman et al, 2015a) and MNLI (Williams et al, 2017) datasets as ppos, and the whole 20000 examples as pdata.…”

Section: Vascl Leads To More Dispersed Representationmentioning

confidence: 99%

“…This issue has motivated more research efforts on designing generic sentence-level learning objectives or tasks. Among them, supervised learning on the Natural Language Inference (NLI) datasets (Bowman et al, 2015a;Williams et al, 2017;Wang et al, 2018) has established benchmark transfer learning performance on various downstream tasks (Conneau et al, 2017;Cer et al, 2018;Reimers and Gurevych, 2019a;. Despite promising progress, the high cost of collecting annotations precludes its wide applicability, especially in the scenario where the target domain has scarce annotations, but differs significantly from the NLI dataset .…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Virtual Augmentation Supported Contrastive Learning of Sentence Representations

Zhang¹,

Xiao²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain specific knowledge. This challenge is magnified in natural language processing where no general rules exist for data augmentation due to the discrete nature of natural language. We tackle this challenge by presenting a Virtual augmentation Supported Contrastive Learning of sentence representations (VaSCL). Originating from the interpretation that data augmentation essentially constructs the neighborhoods of each training instance, we in turn utilize the neighborhood to generate effective data augmentations. Leveraging the large training batch size of contrastive learning, we approximate the neighborhood of an instance via it's K-nearest in-batch neighbors in the representation space. We then define an instance discrimination task within this neighborhood, and generate the virtual augmentation in an adversarial training manner. We access the performance of VaSCL 1 on a wide range of downstream tasks, and set a new state-of-the-art for unsupervised sentence representation learning.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Vascl Leads To More Dispersed Representationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Virtual Augmentation Supported Contrastive Learning of Sentence Representations

Zhang¹,

Xiao²,

Zhu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Language is a highly general and multi-faceted domain, and benchmarks are playing an increasingly important role in NLP: the General Language Understanding Evaluation benchmark (GLUE) contributing to highlight the potential of self-supervised learning for language problems [58,14]. GLUE encompasses 9 different Language classification tasks including sentiment analysis [78], sentence similarity [15,7], Natural Language Inference [12,83], Question Answering [62], and coreference [37]. While state-of-the-art approaches have now achieve a human-level performance on GLUE [42,38,61], Wang et al [82] proposed a more challenging successor, SuperGLUE.…”

Section: Benchmarksmentioning

confidence: 99%

BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Scialom,

Hill

2021

Preprint

View full text Add to dashboard Cite

Natural language processing (NLP) systems are increasingly trained to generate open-ended text rather than classifying between responses. This makes research on evaluation metrics for generated language -functions that score system output given the context and/or human reference responses -of critical importance. However, different metrics have different strengths and biases, and reflect human intuitions better on some tasks than others. There is currently no simple, unified way to compare, analyse or evaluate metrics across a representative set of tasks. Here, we describe the Benchmark to Evaluate Automatic Metrics (BEAMetrics), a resource to make research into new metrics itself easier to evaluate. BEAMetrics users can quickly compare existing and new metrics with human judgements across a diverse set of tasks, quality dimensions (fluency vs. coherence vs. informativeness etc), and languages. As generation experts might predict, BEAMetrics reveals stark taskdependent differences between existing metrics, and consistently poor performance on tasks with complex answer spaces or high reliance on general knowledge. While this analysis highlights a critical issue facing current research practice, BEAMetrics also contribute to its resolution by facilitating research into better metrics -particularly those that can account for the complex interaction between context and general knowledge inherent to many modern NLP applications. 1

show abstract

“…The evolution of deep learning methodologies and continual expansion of computing capacity has enabled many advancements in language modeling field. In specific, transformer-based architectures have foreshadowed the creation of BERT and its many variants, surpassing previously held records in GLUE, SQuAD, and MultiNLI benchmarks [1][2][3][4]. BERT-based architectures have become lightweight and more efficient (DistilBERT) and trained more effectively to become increasingly performant (RoBERTa) [5,6].…”

Section: Introductionmentioning

confidence: 99%

Interpreting Language Models Through Knowledge Graph Extraction

Swamy¹,

Romanou²,

Jäggi³

2021

Preprint

View full text Add to dashboard Cite

Transformer-based language models trained on large text corpora have enjoyed immense popularity in the natural language processing community and are commonly used as a starting point for downstream tasks. While these models are undeniably useful, it is a challenge to quantify their performance beyond traditional accuracy metrics. In this paper, we compare BERT-based language models through snapshots of acquired knowledge at sequential stages of the training process. Structured relationships from training corpora may be uncovered through querying a masked language model with probing tasks. We present a methodology to unveil a knowledge acquisition timeline by generating knowledge graph extracts from cloze "fill-in-the-blank" statements at various stages of RoBERTa's early training. We extend this analysis to a comparison of pretrained variations of BERT models (DistilBERT, BERT-base, RoBERTa). This work proposes a quantitative framework to compare language models through knowledge graph extraction (GED, Graph2Vec) and showcases a part-of-speech analysis (POSOR) to identify the linguistic strengths of each model variant. Using these metrics, machine learning practitioners can compare models, diagnose their models' behavioral strengths and weaknesses, and identify new targeted datasets to improve model performance.

show abstract

A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Cited by 228 publications

References 0 publications

Virtual Augmentation Supported Contrastive Learning of Sentence Representations

Virtual Augmentation Supported Contrastive Learning of Sentence Representations

BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation

Interpreting Language Models Through Knowledge Graph Extraction

Contact Info

Product

Resources

About