ParsiNLU: A Suite of Language Understanding Challenges for Persian

Abstract

Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English. This work focuses on Persian language, one of the widely spoken languages in the world, and yet there are few NLU datasets available for this language. The availability of high-quality evaluation datasets is a necessity for reliable assessment of the progress on different NLU tasks and domains. We introduce ParsiNLU, the first benchmark in Persian language that includes a range of language understanding tasks—reading comprehension, textual entailment, and so on. These datasets are collected in a multitude of ways, often involving manual annotations by native speakers. This results in over 14.5k new instances across 6 distinct NLU tasks. Additionally, we present the first results on state-of-the-art monolingual and multilingual pre-trained language models on this benchmark and compare them with human performance, which provides valuable insights into our ability to tackle natural language understanding challenges in Persian. We hope ParsiNLU fosters further research and advances in Persian language understanding.1

ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization

“…In order to study if ARMAN works well as a language model, we tested our models in Natural Language Understanding (NLU) tasks. According to Khashabi et al (2020), we selected multiple-choice question-answering, textual entailment, sentiment analysis, and question paraphrasing tasks to examine our models' performance on them. For more information about these tasks and datasets, see Appendix A and Khashabi et al (2020).…”

“…ParsiNLU (Khashabi et al, 2020) is a collection of NLU tasks for the Persian language including Textual Entailment, Sentiment Analysis, Question Paraphrasing, Multiple Choice Question Answering, and Reading Comprehension tasks. We have fine-tuned our models on most of them to test their performances on NLU tasks.…”

PersianQuAD: The Native Question Answering Dataset for the Persian Language

“…Hence, in this paper, we create a native QA dataset for the Persian language. Khashabi et al [34] created a Persian QA dataset containing 1300 instances and trained a QA system using this dataset. To the best of our knowledge, currently, there is no native larg-scale QA dataset for answering the Persian questions, neither as a monolingual nor as a cross-lingual dataset.…”

“…[7] 2018 English Native 150K+ Wikiqa: A challenge dataset for open-domain question answering [8] 2015 English Native 3K+ MS MARCO: A human generated machine reading comprehension dataset [9] 2016 English Native 100K+ Natural questions: a benchmark for question answering research [10] 2019 English Native 300K+ Quac: Question answering in context [11] 2018 English Native 100K+ Coqa: A conversational question answering challenge [12] 2019 English Native 127K+ Newsqa: A machine comprehension dataset [13] 2016 English Native 100K+ Constructing datasets for multi-hop reading comprehension across documents [15] 2018 English Native, Multi-hop 50K+ Hotpotqa: A dataset for diverse, explainable multi-hop question answering [16] 2018 English Native, Multi-hop 113K+ Repartitioning of the complexwebquestions dataset [17] 2018 English Native, Multi-hop 63K+ R4C: A benchmark for evaluating RC systems to get the right answer for the right reason [18] 2019 English Native, Multi-hop 4K+ Automatic spanish translation of the squad dataset for multilingual question answering [19] 2019 Spanish Translation 100K+ Neural arabic question answering [20] 2019 Arabic Translation 48K+ Semi-supervised training data generation for multilingual question answering [21] 2018 Korean Translation 81K+ Neural learning for question answering in italian [22] 2018 Italian Translation 60K+ SberQuAD-Russian reading comprehension dataset: Description and analysis [24] 2020 Russian Native 50K+ Drcd: a chinese machine reading comprehension dataset [25] 2018 Chinese Native 30K+ Korquad1. 0: Korean qa dataset for machine reading comprehension [26] 2018 Korean Native 70K+ Project PIAF: Building a Native French Question-Answering Dataset [27] 2020 French Native 3K+ Parsinlu: a suite of language understanding challenges for persian [34] 2021 Persian Native 1K+ ParSQuAD: Persian Question Answering Dataset based on Machine Translation of SQuAD 2.0 [33] 2021 Persian Translation 25K, 70K…”

“…Content may change prior to final publication. ParSQuAD (manual) [33] Persian ALBERT-FA 48.11% 51.66% ParSQuAD (manual) [33] Persian ParsBERT 46.32% 50.06% ParSQuAD (manual) [33] Persian MBERT 52.86% 56.66% ParSQuAD (automatic) [33] Persian ALBERT-FA 64.71% 67.59% ParSQuAD (automatic) [33] Persian ParsBERT 62.42% 65.26% ParSQuAD (automatic) [33] Persian MBERT 67.73% 70.84% ParsiNLU [34] Persian ParsBERT -40.70% ParsiNLU [34] Persian MBERT -49.70% SQuAD [2] English Bert-base 80.8% 88.5% SQuAD [2] English Bert-large 84.1% 90.9% SQuAD-es [19] Spanish MBERT 48.3% 68.1% SberQuAD [24] Russian BERT 66.6% 84.8% KorQuAD [26] Korean BERT 71.68% 89.76% PIAF [27] French CamemBert -79.64%…”