An Oblivious Approach to Machine Translation Quality Estimation

Abstract

Machine translation (MT) is being used by millions of people daily, and therefore evaluating the quality of such systems is an important task. While human expert evaluation of MT output remains the most accurate method, it is not scalable by any means. Automatic procedures that perform the task of Machine Translation Quality Estimation (MT-QE) are typically trained on a large corpus of source–target sentence pairs, which are labeled with human judgment scores. Furthermore, the test set is typically drawn from the same distribution as the train. However, recently, interest in low-resource and unsupervised MT-QE has gained momentum. In this paper, we define and study a further restriction of the unsupervised MT-QE setting that we call oblivious MT-QE. Besides having no access no human judgment scores, the algorithm has no access to the test text’s distribution. We propose an oblivious MT-QE system based on a new notion of sentence cohesiveness that we introduce. We tested our system on standard competition datasets for various language pairs. In all cases, the performance of our system was comparable to the performance of the non-oblivious baseline system provided by the competition organizers. Our results suggest that reasonable MT-QE can be carried out even in the restrictive oblivious setting.

Linguistic Mathematical Relationships Saved or Lost in Translating Texts: Extension of the Statistical Theory of Translation and Its Application to the New Testament

“…Of course, human evaluation can only be done by experts, therefore it is an expensive and time-consuming activity. To avoid this cost, it is necessary to develop mathematical algorhitms which approximate human judgment [3].…”

“…hence take chapter 23, followed by chapter 3, chapter 16 etc. The text of a chapter can appear twice (with probability 1/28 2 ), three times (with probability 1/28 3 ), et cetera, and the new Matthew can contain a number of words greater or smaller than the original text, on the average (the differences are small and do not affect the results).…”

Multiple Communication Channels in Literary Texts

“…Of course, human evaluation can only be done by experts; therefore it is an expensive and time-consuming activity. To avoid this cost, it is necessary to develop mathematical algorithms which approximate human judgment [27]. The theory developed in [1] [2] [3] and the advances presented in the present paper can set some benchmarks for assessing translation quality.…”