Comparative Human and Automatic Evaluation of Glass-Box and Black-Box Approaches to Interactive Translation Prediction

Abstract

Interactive translation prediction (ITP) is a modality of computer-aided translation that assists professional translators by offering context-based computer-generated continuation suggestions as they type. While most state-of-the-art ITP systems follow a glass-box approach, meaning that they are tightly coupled to an adapted machine translation system, a black-box approach which does not need access to the inner workings of the bilingual resources used to generate the suggestions has been recently proposed in the literature: this new approach allows new sources of bilingual information to be included almost seamlessly. In this paper, we compare for the first time the glass-box and the black-box approaches by means of an automatic evaluation of translation tasks between related languages such as English-Spanish and unrelated ones such as Arabic-English and English-Chinese, showing that, with our setup, 20%-50% of keystrokes could be saved using either method and that the black-box approach outperformed the glass-box one in five out of six scenarios operating under similar conditions. We also performed a preliminary human evaluation of English to Spanish translation for both approaches. On average, the evaluators saved 10% keystrokes and were 4% faster with the black-box approach, and saved 15% keystrokes and were 12% slower with the glass-box one; but they could have saved 51% and 69% keystrokes respectively if they had used all the compatible suggestions. Users felt the suggestions helped them to translate faster and easier. All the tools used to perform the evaluation are available as free/open-source software.

Interactive machine translation for the language modernization and spelling normalization of historical documents

“…The IMT framework was introduced during the Tran-sType project [17] and was further developed during Tran-sType2 [4]. New contributions to this framework include developing new generations of the suffix [56] and profiting from the use of the mouse [45]. Marie et al [32] introduced a touch-based interaction to iteratively improve translation quality.…”

Improving the Translation Environment for Professional Translators

“…copy/paste, drag-and-drop within editor). [5][6][7] show that reuse of sub-segments is possible through interactive translation prediction (ITP), a method in which users are presented, as they type, with translation suggestions from all available resources.…”

“…The score to the right of the MT output (0.56) corresponds to the predicted sentence-level quality, which is based on the output of word-level error detection systems and additional sentence-level features, calculated as 1 − TER. 7 As illustrated in this figure, the SCATE platform not only highlights the type and location of errors in a given MT output but also uses this information to predict its sentence-level quality.…”

Improving the Translation Environment for Professional Translators

“…Various surveys and field studies [46,[80][81][82][83][84][85] investigating human-computer interaction, show that translators value improved translation memory (TM)-machine translation (MT) integration methods (e.g., copy/paste, drag-and-drop within editor). References [86][87][88] show that reuse of sub-segments is possible through interactive translation prediction (ITP), a method in which users are presented, as they type, with translation suggestions from all available resources.…”