Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation

Abstract

In NMT we search for the mode of the model distribution to form predictions. The mode and other high-probability translations found by beam search have been shown to often be inadequate in a number of ways. This prevents improving translation quality through better search, as these idiosyncratic translations end up selected by the decoding algorithm, a problem known as the beam search curse. Recently, an approximation to minimum Bayes risk (MBR) decoding has been proposed as an alternative decision rule that would likely not suffer from the same problems. We analyse this approximation and establish that it has no equivalent to the beam search curse. We then design approximations that decouple the cost of exploration from the cost of robust estimation of expected utility. This allows for much larger hypothesis spaces, which we show to be beneficial. We also show that mode-seeking strategies can aid in constructing compact sets of promising hypotheses and that MBR is effective in identifying good translations in them. We conduct experiments on three language pairs varying in amounts of resources available: English into and from German, Romanian, and Nepali. 1

As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning

“…For MBR, we generate 1024 samples per segment using epsilon sampling and re-use the same samples as references. While this approach does not guarantee that the estimation of the expected utility is unbiased (Eikema and Aziz, 2022), it has empirically been found to work well (Freitag et al, 2023).…”

“…A line of research has focused on improving the efficiency of sampling-based MBR. Eikema and Aziz (2022) propose coarse-to-fine MBR, which prunes the hypotheses based on a cheaper metric, and N-by-S MBR, which uses fewer references than hypotheses. Cheng and Vlachos (2023) propose confidence-based pruning, where the number of hypotheses is iteratively reduced based on an increasing number of references.…”

“…Figure 1: How accurately do MBR efficiency methods approximate standard MBR? In this validation experiment on newstest21, we gradually increase efficiency by using fewer references for pairwise utility estimation -either by subsampling the references (N-by-S; Eikema and Aziz, 2022) or by aggregating their representations using partial aggregation (Section 3.3). We report top-20 accuracy, which describes how often an efficiency method ranks the correct hypothesis (as selected by standard MBR) among the top 20 hypotheses.…”

“…Analogously to coarse-to-fine MBR (Eikema and Aziz, 2022), we evaluate an aggregate-to-fine MBR approach. Specifically, we use the aggregate reference to prune the number of hypotheses to 20 in a first step.…”

“…Estimating utility through MC sampling has quadratic complexity in the number of samples, which limits practical application. Previous work suggested pruning the number of samples based on a cheaper metric or a smaller number of references (Eikema and Aziz, 2022;Cheng and Vlachos, 2023). In this paper, we propose reference aggregation, an alternative efficiency technique that exploits the fact that most common metrics represent text sequences in averageable form, e.g., as n-gram statistics or as embeddings.…”