Opportunities for Human-centered Evaluation of Machine Translation Systems

Abstract

Machine translation models are embedded in larger user-facing systems. Although model evaluation has matured, evaluation at the systems level is still lacking. We review literature from both the translation studies and HCI communities about who uses machine translation and for what purposes. We emphasize an important difference in evaluating machine translation models versus the physical and cultural systems in which they are embedded. We then propose opportunities for improved measurement of user-facing translation systems. We pay particular attention to the need for design and evaluation to aid engendering trust and enhancing user agency in future machine translation systems.

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text

“…This presentation, alongside the anthropomorphic bias of deep learning models (Watson, 2019), can perpetuate these opinions including harmful stereotypes. This is a general limitation of NLG models which we are unable to capture using standardized benchmarks alongside intrinsic evaluations, and others have thus called for more work to evaluate models in the physical and cultural context in which they are applied (Liebling et al, 2022;Bhatt et al, 2022). We also note that few, if any, benchmark currently reports the environmental side-effects of training and serving NLG models (Strubell, Ganesh, & Mc-Callum, 2019).…”