An Empirical Study of Generation Order for Machine Translation

Abstract

In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft orderreward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this framework to explore a large variety of generation orders, including uninformed orders, locationbased orders, frequency-based orders, contentbased orders, and model-based orders. Curiously, we find that for the WMT'14 English → German and WMT'18 English → Chinese translation tasks, order does not have a substantial impact on output quality. Moreover, for English → German, we even discover that unintuitive orderings such as alphabetical and shortest-first can match the performance of a standard Transformer, suggesting that traditional left-to-right generation may not be necessary to achieve high performance.

Imputer: Sequence Modelling via Imputation and Dynamic Programming

“…Insertion Transformer has been shown to generate sequences in logarithmic log n iterations (Stern et al, 2019), where as the Imputer requires constant generation steps. Additionally, the Insertion Transformer with its vastly non-monotonic generation order (Chan et al, 2019c) lacks the monoticity that the Imputer dynamic programming endows, consequently we found the Insertion Transformer difficult to apply to speech recognition applications.…”

“…Recently, there has been a growing interest in models that make a trade-off between the two extremes of fully autoregressive and fully non-autoregressive generation, such as the Insertion Transformer (Stern et al, 2019), Mask-Predict (Ghazvininejad et al, 2019), Levenstein Transformer (Gu et al, 2019b) and Multilingual KER-MIT (Chan et al, 2019a). Such models sacrifice almost no performance, while requiring a logarithmic (Chan et al, 2019c) or a constant number of generation steps (Lee et al, 2018).…”

Non-Autoregressive Machine Translation with Latent Alignments

“…There has been significant prior work on nonautoregressive iterative methods for machine translation (Gu et al, 2018), some of which are: iterative refinement (Lee et al, 2018), insertionbased methods Chan et al, 2019a;Li and Chan, 2019), and conditional masked language models (Ghazvininejad et al, 2019(Ghazvininejad et al, , 2020b. Like insertion-based models Chan et al, 2019c), our work does not commit to a fixed target length; insertionbased models can dynamically grow the canvas size, whereas our work which relies on a latent alignment can only generate a target sequence up to a fixed maximum predetermined length. Compared to conditional masked languages models (Ghazvininejad et al, 2019(Ghazvininejad et al, , 2020b, key differences are: 1) our models do not require target length prediction, and 2) we eschew the encoderdecoder neural architecture formulation, but rather rely on the single simple decoder architecture.…”

Towards More Efficient Insertion Transformer with Fractional Positional Encoding

“…The generation is sometimes not fluent, as multiple tokens may compete for the same meaning. The insertion-based methods (Stern et al, 2019;Gu et al, 2019a;Welleck et al, 2019;Chan et al, 2020;Zhang et al, 2020b) also change the standard left-to-right generation by allowing dynamically inserting tokens for the generation process. This provides a good balance between generation fluency and efficiency, and does not require predicting target lengths first.…”

“…For machine translation, we utilize the widely used WMT 2014 English-German translation dataset (Bojar et al, 2014), with newstest2013 as the development and new-stest2014 as the test set. Following previous work (Stern et al, 2019;Chan et al, 2020), we apply sequence-level knowledge distillation (Hinton et al, 2015;Kim and Rush, 2016) from a left-toright autoregressive model, which has been found helpful to reduce data complexity and improve the performance of NAT models (Zhou et al, 2020).…”