VLSP 2021 - ViMRC Challenge: Vietnamese Machine Reading Comprehension

Nguyen, Kiet Van; Tran, Son Quoc; Nguyen, Luan Thanh; Huynh, Tin Van; Luu, Son T.; Nguyen, Ngan Luu-Thuy

doi:10.48550/arxiv.2203.11400

Cited by 3 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, in order to analyze the performances of models on different language aspects, we annotate each unanswerable question into one of 7 unanswerable types, most of which are inspired by (Nguyen et al, 2022).…”

Section: Analysis Methodsmentioning

confidence: 99%

“…Unanswerable questions in MRC draw much attention from the research community after the publication of SQuAD 2.0 (Rajpurkar et al, 2018). Following the guidelines proposed by Rajpurkar et al (2018), unanswerable questions in MRC are introduced in MRC of other languages such as French in FQuAD 2.0 (Heinrich et al, 2021) and Vietnamese in UIT-ViQuAD 2.0 (Nguyen et al, 2022). The research community commonly refers to unanswerable questions in SQuAD, FQuAD, and UIT-ViQuAD as "artificial unanswerable questions" because annotators are instructed to intentionally create questions that cannot be answered using the information provided in the given context.…”

Section: Related Workmentioning

confidence: 99%

“…For example, RoBERTa (Liu et al, 2019), CamemBERT (Martin et al, 2020) and GELECTRA (Chan et al, 2020) achieve near human performances on SQuAD (Rajpurkar et al, 2018), FQuAD (d'Hoffschmidt et al, 2020Heinrich et al, 2021) and GermanQuAD (Möller et al, 2021), respectively. However, for other low-resource languages, such as Vietnamese, the performances of pre-trained language models are significant far lower than that of humans (Nguyen et al, 2022). We can explain these difficulties in research by the underdevelopment of Vietnamese monolingual language models.…”

Section: Introductionmentioning

confidence: 96%

“…Besides, to fully understand the given context, MRC models are expected to acquire extraordinary Reading Comprehension skills such as coreference resolution and bridging, which are part of the multisentence level aspects of language understanding. We focus our analysis on unanswerable questions because unanswerable questions proposed by Nguyen et al (2022) are much more challenging than answerable questions in the same dataset, which directly creates more materials for us to reveal the language weaknesses of models. Additionally, since Nguyen et al (2022) proposed a novel method for annotating unanswerable questions, which involves instructing annotators to use various techniques to transform answerable questions into unanswerable ones instead of generating unanswerable questions from scratch, UIT-ViQuAD 2.0 has successfully introduced many new types of unanswerable questions.…”

Section: Introductionmentioning

confidence: 99%

“…We focus our analysis on unanswerable questions because unanswerable questions proposed by Nguyen et al (2022) are much more challenging than answerable questions in the same dataset, which directly creates more materials for us to reveal the language weaknesses of models. Additionally, since Nguyen et al (2022) proposed a novel method for annotating unanswerable questions, which involves instructing annotators to use various techniques to transform answerable questions into unanswerable ones instead of generating unanswerable questions from scratch, UIT-ViQuAD 2.0 has successfully introduced many new types of unanswerable questions. Therefore, we have a more diverse range of language aspects to analyze the performances of models on.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension

Tran¹,

Do²,

Nguyen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Although the curse of multilinguality significantly restricts the language abilities of multilingual models in monolingual settings, researchers now still have to rely on multilingual models to develop state-of-the-art systems in Vietnamese Machine Reading Comprehension. This difficulty in researching is because of the limited number of high-quality works in developing Vietnamese language models. In order to encourage more work in this research field, we present a comprehensive analysis of language weaknesses and strengths of current Vietnamese monolingual models using the downstream task of Machine Reading Comprehension. From the analysis results, we suggest new directions for developing Vietnamese language models. Besides this main contribution, we also successfully reveal the existence of artifacts in Vietnamese Machine Reading Comprehension benchmarks and suggest an urgent need for new high-quality benchmarks to track the progress of Vietnamese Machine Reading Comprehension. Moreover, we also introduced a minor but valuable modification to the process of annotating unanswerable questions for Machine Reading Comprehension from previous work. Our proposed modification helps improve the quality of unanswerable questions to a higher level of difficulty for Machine Reading Comprehension systems to solve.

show abstract

Section: Analysis Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension

Tran¹,

Do²,

Nguyen³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models

Tran,

Do,

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

Pretrained language models have achieved super-human performances on many Machine Reading Comprehension (MRC) benchmarks. Nevertheless, their relative inability to defend against adversarial attacks has spurred skepticism about their natural language understanding. In this paper, we ask whether training with unanswerable questions in SQuAD 2.0 can help improve the robustness of MRC models against adversarial attacks. To explore that question, we fine-tune three state-of-theart language models on either SQuAD 1.1 or SQuAD 2.0 and then evaluate their robustness under adversarial attacks. Our experiments reveal that current models fine-tuned on SQuAD 2.0 do not initially appear to be any more robust than ones fine-tuned on SQuAD 1.1, yet they reveal a measure of hidden robustness that can be leveraged to realize actual performance gains. Furthermore, we find that the robustness of models fine-tuned on SQuAD 2.0 extends to additional out-of-domain datasets. Finally, we introduce a new adversarial attack to reveal artifacts of SQuAD 2.0 that current MRC models are learning. 1

show abstract

Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension

Tran,

Do,

Nguyen

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research W

View full text Add to dashboard Cite

show abstract

VLSP 2021 - ViMRC Challenge: Vietnamese Machine Reading Comprehension

Cited by 3 publications

References 0 publications

Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension

Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension

The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models

Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension

Contact Info

Product

Resources

About