DaNetQA: a yes/no Question Answering Dataset for the Russian Language

Глушкова, Т. С.; Machnev, Alexey; Fenogenova, Alena; Shavrina, Tatiana; Artemova, Ekaterina; Ignatov, Dmitry I.

doi:10.48550/arxiv.2010.02605

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Originally, DaNetQA had a limited number of examples: 392, 295, 295 (train/val/test). We extended the dataset following the methodology described in [9], and converted a subset of MuSeRC into the yes/no QA setting, labeled by crowd-workers afterward. The new task contains 1750, 821, and 805 examples (train/val/test).…”

Section: Danetqamentioning

confidence: 99%

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models

Fenogenova¹,

Tikhonova²,

Mikhailov³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In the last year, new neural architectures and multilingual pre-trained models have been released for Russian, which led to performance evaluation problems across a range of language understanding tasks.This paper presents Russian SuperGLUE 1.1, an updated benchmark styled after GLUE for Russian NLP models. The new version includes a number of technical, user experience and methodological improvements, including fixes of the benchmark vulnerabilities unresolved in the previous version: novel and improved tests for understanding the meaning of a word in context (RUSSE) along with reading comprehension and common sense reasoning (DaNetQA, RuCoS, MuSeRC). Together with the release of the updated datasets, we improve the benchmark toolkit based on jiant framework for consistent training and evaluation of NLP-models of various architectures which now supports the most recent models for Russian. Finally, we provide the integration of Russian SuperGLUE with a framework for industrial evaluation of the open-source models, MOROCCO (MOdel ResOurCe COmparison), in which the models are evaluated according to the weighted average metric over all tasks, the inference speed, and the occupied amount of RAM. Russian SuperGLUE is publicly available at https: //russiansuperglue.com/.

show abstract

Section: Danetqamentioning

confidence: 99%

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models

Fenogenova¹,

Tikhonova²,

Mikhailov³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the third place we have Russian, which a version of SQuAD [75], a dataset for open-domain QA over Wikidata [134], a boolean QA dataset [91], and datasets for cloze-style commonsense reasoning and multi-choice, multi-hop RC [81].…”

Section: Monolingual Resourcesmentioning

confidence: 99%

QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

Rogers¹,

Gardner²,

Augenstein³

2021

Preprint

View full text Add to dashboard Cite

Alongside huge volumes of research on deep learning models in NLP in the recent years, there has been also much work on benchmark datasets needed to track modeling progress. Question answering and reading comprehension have been particularly prolific in this regard, with over 80 new datasets appearing in the past two years. This study is the largest survey of the field to date. We provide an overview of the various formats and domains of the current resources, highlighting the current lacunae for future work. We further discuss the current classifications of "reasoning types" in question answering and propose a new taxonomy. We also discuss the implications of over-focusing on English,

show abstract

DaNetQA: a yes/no Question Answering Dataset for the Russian Language

Cited by 2 publications

References 0 publications

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models

QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension

Contact Info

Product

Resources

About