Unbabel's Participation in the WMT17 Translation Quality Estimation Shared Task

Abstract

This paper presents the contribution of the Unbabel team to the WMT 2017 Shared Task on Translation Quality Estimation. We participated on the word-level and sentence-level tracks. We describe our two submitted systems: (i) STACKEDQE, a "pure" QE system, trained only on the provided training sets, which is a stacked combination of a feature-rich sequential linear model with a neural network, and (ii) FULLSTACKEDQE, which also stacks the predictions of an automatic post-editing system, trained on additional data. When evaluated on the EnglishGerman and German-English datasets, FULLSTACKEDQE achieved word-level F MULT 1 scores of 56.6% and 52.9%, and sentence-level correlation Pearson scores of 64.1% and 62.6%, respectively. Our system ranked second in both tracks, being statistically indistinguishable from the best system in the word-level track.

“Bilingual Expert” Can Find Translation Errors

“…The sentence level results of WMT 2017 are listed in Table 1. We mainly compared our single model with the two algorithms (Kim et al 2017;Martins, Kepler, and Monteiro 2017), ranking top 3 in the WMT 2017 finalist. Unbabel is combination of a feature-rich sequential linear model with a neural network.…”

Estimating word-level quality of statistical machine translation output using monolingual information alone

“…2016; Scarton et al . 2016; Kim, Lee, and Na 2017; Hokamp 2017; Martins, Kepler, and Monteiro 2017).…”

“…(2017) advanced this model by applying a stack propagation method by allowing backpropagation from the quality estimator to the word predictor and achieved improvements in QE performance. Martins, Kepler, and Monteiro (2017) proposed a word-level QE system which consists of a neural model stacked into a linear feature-rich classifier. In this system, they use a number of features that depend on the context of a given target word and its aligned source word, as well as syntactic features that provide information about the dependency structure of a given sentence.…”

“…Moreover, Martins et al . (2017) proposed using an APE system for predicting word-level quality labels, which relies on a large amount of artificially generated MT–PE sentence pairs, also referred to as “roundtrip translations.” The idea of using such artificial data was a major contribution toward improving APE systems (Junczys-Dowmunt and Grundkiewicz 2016). Hokamp (2017) achieved the state-of-the-art QE performance by using a similar APE approach proposed by by Martins et al .…”

“…Hokamp (2017) achieved the state-of-the-art QE performance by using a similar APE approach proposed by by Martins et al . (2017), by building ensembles of specialized NMT systems for the APE and word-level QE tasks while additionally incorporating syntactic features as “factors” in the input to NMT systems. Similar to the work of Martins et al .…”