PheMT: A Phenomenon-wise Dataset for Machine Translation Robustness on User-Generated Contents

Abstract

Neural Machine Translation (NMT) has shown drastic improvement in its quality when translating clean input, such as text from the news domain. However, existing studies suggest that NMT still struggles with certain kinds of input with considerable noise, such as User-Generated Contents (UGC) on the Internet. To make better use of NMT for cross-cultural communication, one of the most promising directions is to develop a model that correctly handles these expressions. Though its importance has been recognized, it is still not clear as to what creates the great gap in performance between the translation of clean input and that of UGC. To answer the question, we present a new dataset, PheMT, for evaluating the robustness of MT systems against specific linguistic phenomena in Japanese-English translation. Our experiments with the created dataset revealed that not only our in-house models but even widely used off-the-shelf systems are greatly disturbed by the presence of certain phenomena.

Noisy UGC Translation at the Character Level: Revisiting Open-Vocabulary Capabilities and Robustness of Char-Based Models

“…In line to Fujii et al (2020) findings, where finegrained NMT granularity provided robustness advantages when processing misspelling, our results show that the best and worst translations' specificities distribution point to a better performance of char2char for the missing diacritics category, giving insights on more specific types of misspellings that affect performance.…”

RoCS-MT: Robustness Challenge Set for Machine Translation

“…Several parallel UGC datasets exist across different language pairs. While some are extracted automatically from crawled data (Ling et al, 2013;Vicente et al, 2016;Mubarak et al, 2020), a majority are based on monolingual sentences that are then translated into the target language (Sluyter-Gäthje et al, 2018;Michel and Neubig, 2018;Rosales Núñez et al, 2019;Fujii et al, 2020;McNamee and Duh, 2022). The closest to our RoCS-MT dataset are (Michel and Neubig, 2018) and (Rosales Núñez et al, 2019), which were designed to contain challenging non-standard phenomena, whereas many of the existing datasets do not apply any such filter.…”

Understanding the Impact of UGC Specificities on Translation Quality

“…Several UGC parallel corpora (Michel and Neubig, 2018a;Rosales Núñez et al, 2019) have been introduced to evaluate the robustness of MT, some of which, such as (Fujii et al, 2020), are specially annotated to identify UGC idiosyncrasies allowing to measure the impact of a given specificity. Our analyses ( §2), indeed, show that measuring the average-case performance using a standard metric on a UGC test set falls far short of giving a reliable image of the UGC translation quality: explaining the observed performance gap requires a particular evaluation framework made of tailored metrics and specific test sets in which UGC idiosyncrasies have been precisely annotated.…”