Toward More Effective Human Evaluation for Machine Translation

Abstract

Improvements in text generation technologies such as machine translation have necessitated more costly and time-consuming human evaluation procedures to ensure an accurate signal. We investigate a simple way to reduce cost by reducing the number of text segments that must be annotated in order to accurately predict a score for a complete test set. Using a sampling approach, we demonstrate that information from document membership and automatic metrics can help improve estimates compared to a pure random sampling baseline. We achieve gains of up to 20% in average absolute error by leveraging stratified sampling and control variates. Our techniques can improve estimates made from a fixed annotation budget, are easy to implement, and can be applied to any problem with structure similar to the one we study.

Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation

“…Being able to filter out exactly semantically equivalent sentence pairs would reduce this workload. Similarly, filtering out exactly semantically equivalent sentences can lessen the amount of annotation necessary for human evaluations of text (Saldías et al, 2022).…”

A Research-Based Guide for the Creation and Deployment of a Low-Resource Machine Translation System

“…While claims are made by recent literature (Goyal et al, 2022) that a human's involvement is timely and expensive, it cannot be absent. In order to determine acceptable values for human involvement, we rely on the past investigation in the area (Koehn, 2009;González Rubio, 2014;Way, 2018;Kreutzer et al, 2022;Saldías et al, 2022) to answer the main questions below.…”

“…Other work Doherty, 2018) mentions that translation quality assessments around 60 to 70% are acceptable. For a LRMTS, the human involvement can lead to high quality LRMTS as shown by (Saldías et al, 2022).…”