FQuAD: French Question Answering Dataset

Abstract

Recent advances in the field of language modeling have improved state-of-the-art results on many Natural Language Processing tasks. Among them, the Machine Reading Comprehension task has made significant progress. However, most of the results are reported in English since labeled resources available in other languages, such as French, remain scarce. In the present work, we introduce the French Question Answering Dataset (FQuAD). FQuAD is French Native Reading Comprehension dataset that consists of 25,000+ questions on a set of Wikipedia articles. A baseline model is trained which achieves an F1 score of 88.0% and an exact match ratio of 77.9% on the test set. The dataset is made freely available at https://fquad.illuin.tech.1 To our knowledge, this dataset is the first of it's kind although ongoing efforts are being made on the same task at https://piaf.etalab.studio 2 Both contributions are freely available at https: //fquad.illuin.tech.

Project PIAF: Building a Native French Question-Answering Dataset

“…In Table 4 significantly lower (10 absolute F1 points) than that reported by (d'Hoffschmidt et al, 2020) on the FQUAD hidden test set. 19 This can be explained by a number of factors, including: hyperparameters' setup (not reported in the FQUAD paper); the use of additional answers for computing evaluation scores on the hidden test set (although, for reference, this factor only justifies 3-4 points of difference on the SQuAD dev set).…”

“…In (d'Hoffschmidt et al, 2020), the authors rely on Camem-BERT (Martin et al, 2019) for their evaluations, but do not report the hyper-parameters used. For all our experiments, we use batch size = 8, learning rate = 3e −5 , n epochs = 2, max seq len = 384, doc stride = 128.…”

Cedille: A large autoregressive French language model

“…Question answering (QA) was evaluated on FQuAD (French Question Answering Dataset) [30], a dataset inspired by the English SQuAD equivalent [31]. The models were evaluated on the validation subset, which contains 3188 human-curated question-answer pairs, based on 768 high-quality French Wikipedia articles.…”

“…GPT-3 (davinci) was not evaluated on this task for cost reasons, as OpenAI did not support our request for funding at the time of writing. The results may be contrasted to a finetuned version of Camem-BERT [32] which yields F1 of 88% and best match of 78% on this dataset [30].…”

Pirá: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean

“…That poses a problem for many languages; even though a number of datasets are available in English [6,[31][32][33], resources in other languages are rather scarce. While a few other languages have received some attention, such as Chinese [8,18], French [11], and German [27], and while one can find multilingual datasets around [2,23,25], many languages, such as Portuguese, still lag behind. Question answering (QA) in non-English languages suffers from an additional difficulty: many, and in some cases most, of the documents used to answer questions are only available in English.…”