Exploring gap filling as a cheaper alternative to reading comprehension questionnaires when evaluating machine translation for gisting

Abstract

A popular application of machine translation (MT) is gisting: MT is consumed as is to make sense of text in a foreign language. Evaluation of the usefulness of MT for gisting is surprisingly uncommon. The classical method uses reading comprehension questionnaires (RCQ), in which informants are asked to answer professionally-written questions in their language about a foreign text that has been machine-translated into their language. Recently, gap-filling (GF), a form of cloze testing, has been proposed as a cheaper alternative to RCQ. In GF, certain words are removed from reference translations and readers are asked to fill the gaps left using the machine-translated text as a hint. This paper reports, for the first time, a comparative evaluation, using both RCQ and GF, of translations from multiple MT systems for the same foreign texts, and a systematic study on the effect of variables such as gap density, gap-selection strategies, and document context in GF. The main findings of the study are: (a) both RCQ and GF clearly identify MT to be useful; (b) global RCQ and GF rankings for the MT systems are mostly in agreement; (c) GF scores vary very widely across informants, making comparisons among MT systems hard, and (d) unlike RCQ, which is framed around documents, GF evaluation can be framed at the sentence level. These findings support the use of GF as a cheaper alternative to RCQ.

Google Translate in Massive Open Online Courses: Evaluation of Accuracy and Comprehensibility among Female Users

“…Evidently, the machine-translated subtitles were successful in conveying some of the presented information to the participants. The findings of the quantitative analysis are consistent with those of Forcada et al (2018), who reported a lack of any significant differences in reading comprehension between participants who were provided with different MT outputs and those who were provided with human translations. The researchers stated that this result is a clear indication of the usefulness of raw MT output in assimilating the gist of the presented information.…”

Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation

“…As alternatives to adequacy and fluency, presented reading comprehension for MT quality evaluation. Forcada et al (2018) proposed gap-filling, where certain words are removed from reference translations and readers are asked to fill the gaps left using the machine-translated text as a hint. Popović (2020) proposed to ask annotators to just label problematic parts of the translations instead of assigning a score.…”

Evaluer, diagnostiquer et analyser la traduction automatique neuronale

“…La métrique TER (Translation EDIT Rate) de Snover et al [2006] illustre cette démarche et permet d'approximer une mesure du temps de post-édition. Avec les progrès des outils de traitement des langues, la mesure automatique d'autres tâches, en particulier de tâches de compréhension, semble envisageable [Scarton and Specia, 2016, Forcada et al, 2018, Krubi ński et al, 2021.…”