On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation

Abstract

Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual textual similarity. In this paper, we concern ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations, which represents a natural adversarial setup for multilingual encoders. Referencefree evaluation holds the promise of web-scale comparison of MT systems. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations, namely, (a) a semantic mismatch between representations of mutual translations and, more prominently, (b) the inability to punish "translationese", i.e., low-quality literal translations. We propose two partial remedies:(1) post-hoc re-alignment of the vector spaces and (2) coupling of semantic-similarity based metrics with target-side language modeling. In segment-level MT evaluation, our best metric surpasses reference-based BLEU by 5.7 correlation points. We make our MT evaluation code available. 1

COMET: A Neural Framework for MT Evaluation

“…A variation on our HTER estimator model trained with the vector x = [h; s; r; h s; h r; |h − s|; |h − r|] as input to the feed-forward only succeed in boosting segment-level performance in 8 of the 18 language pairs outlined in section 5 below and the average improvement in Kendall's Tau in those settings was +0.0009. As noted in Zhao et al (2020), while cross-lingual pretrained models are adaptive to multiple languages, the feature space between languages is poorly aligned. On this basis we decided in favor of excluding the source embedding on the intuition that the most important information comes from the reference embedding and reducing the feature space would allow the model to focus more on relevant information.…”

Probing Multilingual BERT for Genetic and Typological Signals

“…Overall, the average performance for mBERT (0.16) is five times better than random guessing, but consistently lower than the performance for mFastText (0.46 on average). 7 Overall, this shows that mBERT does not properly capture multilingual semantics, a finding that is echoed in some other recent works Zhao et al, 2020b). The apparent reason lies in its naive training process, which does not exploit cross-lingual signals but merely trains on the concatenation of all languages.…”

“…K et al (2020) show that lexical overlap plays no big role in cross-lingual transfer for mBERT, but the depth of the network does, with deeper models having better transfer. Zhao et al (2020b) find that mBERT lacks fine-grained cross-lingual text understanding and can be fooled by adversarial inputs produced by the corrupt input produced by MT systems.…”

KoBE: Knowledge-Based Machine Translation Evaluation

“…Our method is simple, interpretable and produces scores closer to human judgements on an absolute scale, while enabling more finegrained analysis which can be useful to find weak spots in the evaluated model. In future work, we would like to combine knowledge-based signals with unsupervised approaches like YiSi (Lo, 2019) and XMoverScore (Zhao et al, 2020) that use contextualized representations from cross-lingual LMs like multilingual BERT (Devlin et al, 2019). As our method does not require reference translations, we would like to explore scaling it to use much larger or domain specific monolingual datasets.…”