SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis

Efimov, Pavel; Chertok, Andrey; Boytsov, Leonid; Braslavski, Pavel

doi:10.1007/978-3-030-58219-7_1

Cited by 45 publications

(31 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Moreover, additional training on English science QA in lower school levels has no significant effect on the overall accuracy. These results suggest that further investigation of finetuning with other multilingual datasets (Gupta et al, 2018;Lewis et al, 2020;Efimov et al, 2020;d'Hoffschmidt et al, 2020;Artetxe et al, 2020;Longpre et al, 2020) is needed in order to understand the domain transfer benefits to science QA in Eχαµs, even if they are not in a multi-choice setting (Khashabi et al, 2020). Using domain-adaptive and task-adaptive pre-training (Gururangan et al, 2020) to the multilingual science QA might offer further potential benefits.…”

Section: Discussionmentioning

confidence: 86%

“…Other efforts focused on building bi-lingual datasets that are similar in spirit to SQuAD (Rajpurkar et al, 2016) -extractive reading comprehension over open-domain articles. Such datasets are collected by crowdsourcing questions, following a procedure similar to (Rajpurkar et al, 2016), in Russian (Efimov et al, 2020), Korean (Lim et al, 2019), French (d'Hoffschmidt et al, 2020, or by translating existing English QA pairs to Spanish (Carrino et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering

Hardalov¹,

Mihaylov²,

Zlatkova³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

We propose Eχαµs -a new benchmark dataset for cross-lingual and multilingual question answering for high school examinations. We collected more than 24,000 highquality high school exam questions in 16 languages, covering 8 language families and 24 school subjects from Natural Sciences and Social Sciences, among others.Eχαµs offers a fine-grained evaluation framework across multiple languages and subjects, which allows precise analysis and comparison of various models. We perform various experiments with existing top-performing multilingual pre-trained models and we show that Eχαµs offers multiple challenges that require multilingual knowledge and reasoning in multiple domains. We hope that Eχαµs will enable researchers to explore challenging reasoning and knowledge transfer methods and pretrained models for school question answering in various languages which was not possible before. The data, code, pre-trained models, and evaluation are available at http:// github.com/mhardalov/exams-qa.

show abstract

Section: Discussionmentioning

confidence: 86%

Section: Related Workmentioning

confidence: 99%

EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering

Hardalov¹,

Mihaylov²,

Zlatkova³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…Among them, some initiatives have been carried out in Chinese, Korean and Russian and all of them have been built in a similar way to SQuAD1.1. The SberQuAD dataset (Efimov et al, 2019) is a Russian native Reading Comprehension dataset and is made up of 50,000+ samples. The CMRC 2018 (Cui et al, 2019) dataset is a Chinese native Reading Comprehension dataset that gathers 20,000+ question and answer pairs.…”

Section: Reading Comprehension In Other Languagesmentioning

confidence: 99%

FQuAD: French Question Answering Dataset

d'Hoffschmidt¹,

Belblidia²,

Heinrich³

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Recent advances in the field of language modeling have improved state-of-the-art results on many Natural Language Processing tasks. Among them, Reading Comprehension has made significant progress over the past few years. However, most results are reported in English since labeled resources available in other languages, such as French, remain scarce. In the present work, we introduce the French Question Answering Dataset (FQuAD). FQuAD is a French Native Reading Comprehension dataset of questions and answers on a set of Wikipedia articles that consists of 25,000+ samples for the 1.0 version and 60,000+ samples for the 1.1 version. We train a baseline model which achieves an F1 score of 92.2 and an exact match ratio of 82.1 on the test set. In an effort to track the progress of French Question Answering models we propose a leaderboard and we have made the 1.0 version of our dataset freely available at https://illuin-tech. github.io/FQuAD-explorer/.

show abstract

“…Translated datasets have also been used in making cross lingual benchmark datasets like XQuAD (Artetxe et al, 2019) and MLQA (Lewis et al, 2019). Aside from using translated dataset, there has also been attempts of curating large question answering datasets in multiple other languages including French (d'Hoffschmidt et al, 2020), Korean (Lim et al, 2019), Russian (Efimov et al, 2020), Chinese (Cui et al, 2018;Shao et al, 2018) and benchmark models like QANet (Yu et al, 2018), BiDAF (Seo et al, 2016), BERT (Devlin et al, 2018) have been trained on them. In contrast to gathering translated or human annotated dataset for model training, zero shot transfer learning where pretrained models were evaluated directly on a new language after task specific training on question answering has also been attempted on reading comprehension tasks (Artetxe et al, 2019;Hsu et al, 2019;Siblini et al, 2019).…”

Section: Question Answering In Englishmentioning

confidence: 99%

Deep learning based question answering system in Bengali

Mayeesha

Sarwar

Rahman

2020

Journal of Information and Telecommunication

View full text Add to dashboard Cite

Recent advances in the field of natural language processing has improved state-of-the-art performances on many tasks including question answering for languages like English. Bengali language is ranked seventh and is spoken by about 300 million people all over the world. But due to lack of data and active research on QA similar progress has not been achieved for Bengali. Unlike English, there is no benchmark large scale QA dataset collected for Bengali, no pretrained language model that can be modified for Bengali question answering and no human baseline score for QA has been established either. In this work we use state-of-theart transformer models to train QA system on a synthetic reading comprehension dataset translated from one of the most popular benchmark datasets in English called SQuAD 2.0. We collect a smaller human annotated QA dataset from Bengali Wikipedia with popular topics from Bangladeshi culture for evaluating our models. Finally, we compare our models with human children to set up a benchmark score using survey experiments.

show abstract

SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis

Cited by 45 publications

References 18 publications

EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering

EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering

FQuAD: French Question Answering Dataset

Deep learning based question answering system in Bengali

Contact Info

Product

Resources

About