Error Investigation of Pre-trained BERTology Models on Vietnamese Natural Language Inference

Hevenli: A Large Dataset for English-Vietnamese Cross-Lingual Natural Language Inference in Healthcare

“…These limitations may reduce the effectiveness of models trained on this dataset when applied on other healthcare topics. Nevertheless, they can be addressed by adding more diverse text sources when selecting the premise sentences, or by combining with other Vietnamese monolingual datasets, e.g., the ViNLI dataset [17].…”

“…There are few works for other languages [3,16,23,34], and particularly, for Vietnamese [17]. These works however focus on constructing open-domain datasets and also only for the monolingual problem.…”

“…The table clearly shows that our Hevenli dataset is much larger than the XNLI dataset, and the sentences of the Hevenli dataset are also much longer than those of the XNLI dataset. As previous works have suggested that the length of both premise and hypothesis sentences may affect the difficulty level of the dataset [16,17,33], we further investigated the length of sentences in our dataset. Figure 1 shows the distributions of the length as measured by the number of tokens in the sentences.…”

“…About 83.7% (respectively 60.0%, 74.0% and 74.2%) of the premise sentences (respectively hypothesis sentences of Entailment, Contradiction, and Neutral label) have 15 to 30 tokens. Also previous works have shown that the linguistic overlapping between the premise and the hypothesis sentences may help in predicting the label [17,23,24], we hence examined the overlapping within our dataset. Since our premise sentences are in English while our hypothesis sentences are in Vietnamese, it does not make sense to measure the overlapping directly.…”