An Analysis of Dataset Overlap on Winograd-Style Tasks

Emami, Ali; Suleman, Kaheer; Trischler, Adam; Cheung, Jackie Chi Kit

doi:10.18653/v1/2020.coling-main.515

“…Knowledge leakage in the evaluation. One recent finding in the field is about knowledge leakage between train and evaluation sets (Lewis et al, 2020b;Emami et al, 2020). Similar concerns have motivated our careful train/evaluation splits ( §4) and experiments with varying training set sizes.…”

Section: Discussionmentioning

confidence: 79%

GooAQ: Open Question Answering with Diverse Answer Types

Khashabi¹,

Ng²,

Khot³

et al. 2021

Preprint

View full text Add to dashboard Cite

While day-to-day questions come with a variety of answer types, the current questionanswering (QA) literature has failed to adequately address the answer diversity of questions. To this end, we present GOOAQ, a large-scale dataset with a variety of answer type. This dataset contains with over 5 million questions and 3 million answers collected from Google. GOOAQ questions are collected semi-automatically from the Google search engine using its autocomplete feature. This results in naturalistic questions of practical interest that are nonetheless short and expressed using simple language. GOOAQ answers are mined from Google's responses to our collected questions, specifically from the answer boxes in the search results. This yields a rich space of answer types, containing both textual answers (short and long) as well as more structured ones such as collections. We benchmark T5 models on GOOAQ and observe that: (a) in line with recent work, LM's strong performance on GOOAQ's short-answer questions heavily benefit from annotated data; however, (b) their quality in generating coherent and accurate responses for questions requiring long responses (such as 'how' and 'why' questions) is less reliant on observing annotated data and mainly supported by their pre-training. We release GOOAQ to facilitate further research on improving QA with diverse response types. 1

show abstract

“…So far the fundamental paradigm for NLP work based on machine learning focused on indistribution evaluation: the test sample would come from the same distribution as the train/validation samples, and the samples would be randomly split. Within that paradigm, it is essential that there are no overlaps between training and test data, which is an issue for many current resources (Lewis et al, 2021;Emami et al, 2020).…”

Section: Evaluation Methodologymentioning

confidence: 99%

Changing the World by Changing the Data

Rogers¹

2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

NLP community is currently investing a lot more research and resources into development of deep learning models than training data. While we have made a lot of progress, it is now clear that our models learn all kinds of spurious patterns, social biases, and annotation artifacts. Algorithmic solutions have so far had limited success. An alternative that is being actively discussed is more careful design of datasets so as to deliver specific signals. This position paper maps out the arguments for and against data curation, and argues that fundamentally the point is moot: curation already is and will be happening, and it is changing the world. The question is only how much thought we want to invest into that process.

show abstract

“…So far the fundamental paradigm for NLP work based on machine learning focused on indistribution evaluation: the test sample would come from the same distribution as the train/validation samples, and the samples would be randomly split. Within that paradigm, it is essential that there are no overlaps between training and test data, which is an issue for many current resources (Lewis et al, 2021;Emami et al, 2020).…”

Section: Evaluation Methodologymentioning

confidence: 99%

Changing the World by Changing the Data

Rogers¹

2021

Preprint

View full text Add to dashboard Cite

NLP community is currently investing a lot more research and resources into development of deep learning models than training data. While we have made a lot of progress, it is now clear that our models learn all kinds of spurious patterns, social biases, and annotation artifacts. Algorithmic solutions have so far had limited success. An alternative that is being actively discussed is more careful design of datasets so as to deliver specific signals. This position paper maps out the arguments for and against data curation, and argues that fundamentally the point is moot: curation already is and will be happening, and it is changing the world. The question is only how much thought we want to invest into that process.

show abstract

An Analysis of Dataset Overlap on Winograd-Style Tasks

Cited by 10 publications

References 20 publications

GooAQ: Open Question Answering with Diverse Answer Types

GooAQ: Open Question Answering with Diverse Answer Types

Changing the World by Changing the Data

Changing the World by Changing the Data

Contact Info

Product

Resources

About