DaNetQA: A Yes/No Question Answering Dataset for the Russian Language

Глушкова, Т. С.; Machnev, Alexey; Fenogenova, Alena; Shavrina, Tatiana; Artemova, Ekaterina; Ignatov, Dmitry I.

doi:10.1007/978-3-030-72610-2_4

Cited by 4 publications

(2 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Originally, DaNetQA had a limited number of examples: 392, 295, 295 (train/val/test). We extended the dataset following the methodology described in [9], and converted a subset of MuSeRC into the yes/no QA setting, labeled by crowd-workers afterward. The new task contains 1750, 821, and 805 examples (train/val/test).…”

Section: Danetqamentioning

confidence: 99%

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP-models

Fenogenova¹,

Tikhonova²,

Mikhailov³

et al. 2021

Computational Linguistics and Intellectual Technologies

Self Cite

View full text Add to dashboard Cite

In the last year, new neural architectures and multilingual pre-trained models have been released for Russian, which led to performance evaluation problems across a range of language understanding tasks.This paper presents Russian SuperGLUE 1.1, an updated benchmark styled after GLUE for Russian NLP models. The new version includes a number of technical, user experience and methodological improvements, including fixes of the benchmark vulnerabilities unresolved in the previous version: novel and improved tests for understanding the meaning of a word in context (RUSSE) along with reading comprehension and common sense reasoning (DaNetQA, RuCoS, MuSeRC). Together with the release of the updated datasets, we improve the benchmark toolkit based on jiant framework for consistent training and evaluation of NLP-models of various architectures which now supports the most recent models for Russian. Finally, we provide the integration of Russian SuperGLUE with a framework for industrial evaluation of the open-source models, MOROCCO (MOdel ResOurCe COmparison), in which the models are evaluated according to the weighted average metric over all tasks, the inference speed, and the occupied amount of RAM. Russian SuperGLUE is publicly available at https: //russiansuperglue.com/.

show abstract

Section: Danetqamentioning

confidence: 99%

Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP-models

Fenogenova¹,

Tikhonova²,

Mikhailov³

et al. 2021

Computational Linguistics and Intellectual Technologies

Self Cite

View full text Add to dashboard Cite

show abstract

“…We address the yes/no QA task, that is, questions that can be answered with either a yes or a no. Transformer-based models and transfer learning are the cutting-edge technologies in yes/no QA (Yin et al 2020;Ignatov 2021). The main existing research insights are that: i) adapting pretrained language models to other tasks improves the accuracy in yes/no QA and ii) the higher the similarity of these tasks to yes/no QA, the better the results.…”

Section: Introductionmentioning

confidence: 99%

Artificial fine-tuning tasks for yes/no question answering

Dimitriadis

Tsoumakas

2022

Nat. Lang. Eng.

View full text Add to dashboard Cite

Current research in yes/no question answering (QA) focuses on transfer learning techniques and transformer-based models. Models trained on large corpora are fine-tuned on tasks similar to yes/no QA, and then the captured knowledge is transferred for solving the yes/no QA task. Most previous studies use existing similar tasks, such as natural language inference or extractive QA, for the fine-tuning step. This paper follows a different perspective, hypothesizing that an artificial yes/no task can transfer useful knowledge for improving the performance of yes/no QA. We introduce three such tasks for this purpose, by adapting three corresponding existing tasks: candidate answer validation, sentiment classification, and lexical simplification. Furthermore, we experimented with three different variations of the BERT model (BERT base, RoBERTa, and ALBERT). The results show that our hypothesis holds true for all artificial tasks, despite the small size of the corresponding datasets that are used for the fine-tuning process, the differences between these tasks, the decisions that we made to adapt the original ones, and the tasks’ simplicity. This gives an alternative perspective on how to deal with the yes/no QA problem, that is more creative, and at the same time more flexible, as it can exploit multiple other existing tasks and corresponding datasets to improve yes/no QA models.

show abstract