Distributionally Robust Language Modeling

Oren, Yonatan; Sagawa, Shiori; Hashimoto, Tatsunori; Liang, Percy

doi:10.18653/v1/d19-1432

Cited by 75 publications

(102 citation statements)

References 24 publications

(25 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our data union can be seen as a simple approach toward this end, but we found that current models do not exploit useful information beyond each target dataset. More sophisticated approaches, such as distributionally robust optimization (Delage and Ye, 2010;Oren et al, 2019), may help. Another promising way is relying on strong pretrained language models, including BERT (Devlin et al, 2019).…”

Section: Discussionmentioning

confidence: 99%

An empirical analysis of existing systems and datasets toward general simple question answering

Han¹,

Topić²,

Noji³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

In this paper, we evaluate the progress of our field toward solving simple factoid questions over a knowledge base, a practically important problem in natural language interface to database. As in other natural language understanding tasks, a common practice for this task is to train and evaluate a model on a single dataset, and recent studies suggest that SimpleQuestions, the most popular and largest dataset, is nearly solved under this setting. However, this common setting does not evaluate the robustness of the systems outside of the distribution of the used training data. We rigorously evaluate such robustness of existing systems using different datasets. Our analysis, including shifting of training and test datasets and training on a union of the datasets, suggests that our progress in solving SimpleQuestions dataset does not indicate the success of more general simple question answering. We discuss a possible future direction toward this goal.

show abstract

Section: Discussionmentioning

confidence: 99%

An empirical analysis of existing systems and datasets toward general simple question answering

Han¹,

Topić²,

Noji³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…While large pre-trained models have shown to work well, many questions and challenges remain. Recent work has shown that these models degrade on out-of-domain data, maximum likelihood training makes them too over-confident (Oren et al, 2019) and particularly calibration is important for out-ofdomain generalization (Hendrycks et al, 2020). An acknowledged issue with fine-tuning is the brittleness of the process (Phang et al, 2018;Dodge et al, 2020).…”

Section: Pre-training-mentioning

confidence: 99%

Neural Unsupervised Domain Adaptation in NLP—A Survey

Ramponi¹,

Plank²

2020

Proceedings of the 28th International Conference on Computational Linguistics

148

108

View full text Add to dashboard Cite

Deep neural networks excel at learning from labeled data and achieve state-of-the-art results on a wide array of Natural Language Processing tasks. In contrast, learning from unlabeled data, especially under domain shift, remains a challenge. Motivated by the latest advances, in this survey we review neural unsupervised domain adaptation techniques which do not require labeled target domain data. This is a more challenging yet a more widely applicable setup. We outline methods, from early traditional non-neural methods to pre-trained model transfer. We also revisit the notion of domain, and we uncover a bias in the type of Natural Language Processing tasks which received most attention. Lastly, we outline future directions, particularly the broader need for out-of-distribution generalization of future NLP. 1

show abstract

“…The question-answering dataset SQuAD 2.0 was created in response to the observation that existing systems could not reliably demur when presented with an unanswerable question (Rajpurkar et al, 2016(Rajpurkar et al, , 2018. The perplexity of language models rises when given out-of-domain text (Oren et al, 2019).…”

Section: Robustnessmentioning

confidence: 99%

Utility is in the Eye of the User: A Critique of NLP Leaderboards

Ethayarajh

Jurafsky

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

103

View full text Add to dashboard Cite

Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards -in their current form -can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model's utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).

show abstract

Distributionally Robust Language Modeling

Cited by 75 publications

References 24 publications

An empirical analysis of existing systems and datasets toward general simple question answering

An empirical analysis of existing systems and datasets toward general simple question answering

Neural Unsupervised Domain Adaptation in NLP—A Survey

Utility is in the Eye of the User: A Critique of NLP Leaderboards

Contact Info

Product

Resources

About