Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information

Ran, Qiu; Lin, Yankai; Zhou, Jie

doi:10.48550/arxiv.1911.02215

Cited by 3 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…UniLM (Dong et al 2019;Bao et al 2020) pre-trains encoder based model with three tasks: unidirectional, bidirectional, and prediction, which allow it can be fine-tuned for both natural language understanding and generation tasks. For the encoder-decoder based models (Song et al 2019;Qi et al 2020b;Song et al 2019), these models are pre-trained with sequence-to-sequence tasks to help the down-stream generation tasks.…”

Section: Related Workmentioning

confidence: 99%

“…This also alleviates the multimodality problem (Gu et al 2017) in non-autoregeressive generation. Many non-autoregeressive translation methods are proposed for better alignment, like fertiity (Gu et al 2017), SoftCopy (Wei et al 2019) or adding reordering module (Ran et al 2019). However, the source and target words in monolingual generation tasks cannot be aligned directly like translation.…”

Section: Motivationmentioning

confidence: 99%

“…Pre-trained models, like BERT (Devlin et al 2018), UniLM (Dong et al 2019) and RoBERTa (Liu et al 2019) have been widely applied on natural language process tasks by transferring the knowledge learned from large amount of unlabeled corpus to downstream tasks. Some pre-trained models (Dong et al 2019;Song et al 2019) also have proven effective for natural language generation tasks in autoregressive generation. However, low latency is required by an increasing number of real-world generating applications, such as online query rewriting in search engines, limiting the use of these autoregressive models (Mohankumar, Begwani, and Singh 2021).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Improving Non-autoregressive Generation with Mixup Training

Jiang¹,

Huang²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

While pre-trained language models have achieved great success on various natural language understanding tasks, how to effectively leverage them into nonautoregressive generation tasks remains a challenge. To solve this problem, we present a non-autoregressive generation model based on pre-trained transformer models. To bridge the gap between autoregressive and non-autoregressive models, we propose a simple and effective iterative training method called MIx Source and pseudo Target (MIST). Unlike other iterative decoding methods, which sacrifice the inference speed to achieve better performance based on multiple decoding iterations, MIST works in the training stage and has no effect on inference time. Our experiments on three generation benchmarks including question generation, summarization and paraphrase generation, show that the proposed framework achieves the new state-ofthe-art results for fully non-autoregressive models. We also demonstrate that our method can be used to a variety of pre-trained models. For instance, MIST based on the small pre-trained model also obtains comparable performance with seq2seq models. Our code is available at https://github.com/kongds/MIST.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Motivationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving Non-autoregressive Generation with Mixup Training

Jiang¹,

Huang²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Based on variational inference, Ma et al (2019) proposed FlowSeq to model sequence-to-sequence generation using generative flow, and Shu et al (2020) introduced LaNMT with continuous latent variables and deterministic inference. Bao et al (2019); Ran et al (2019) used the position information as latent variables to explicitly model the reordering information in the decoding procedure.…”

Section: Non-autoregressive Translationmentioning

confidence: 99%

Sequence-Level Training for Non-Autoregressive Neural Machine Translation

Shao¹,

Feng²,

Zhang³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In recent years, Neural Machine Translation (NMT) has achieved notable results in various translation tasks. However, the word-by-word generation manner determined by the autoregressive mechanism leads to high translation latency of the NMT and restricts its low-latency applications. Non-Autoregressive Neural Machine Translation (NAT) removes the autoregressive mechanism and achieves significant decoding speedup through generating target words independently and simultaneously. Nevertheless, NAT still takes the word-level cross-entropy loss as the training objective, which is not optimal because the output of NAT cannot be properly evaluated due to the multimodality problem. In this paper, we propose using sequence-level training objectives to train NAT models, which evaluate the NAT outputs as a whole and correlates well with the real translation quality. Firstly, we propose training NAT models to optimize sequence-level evaluation metrics (e.g., BLEU) based on several novel reinforcement algorithms customized for NAT, which outperforms the conventional method by reducing the variance of gradient estimation. Secondly, we introduce a novel training objective for NAT models, which aims to minimize the Bag-of-Ngrams (BoN) difference between the model output and the reference sentence. The BoN training objective is differentiable and can be calculated efficiently without doing any approximations. Finally, we apply a three-stage training strategy to combine these two methods to train the NAT model. We validate our approach on four translation tasks (WMT14 En↔De, WMT16 En↔Ro), which shows that our approach largely outperforms NAT baselines and achieves remarkable performance on all translation tasks. 12

show abstract

Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation

Shao¹,

Zhang²,

Feng³

et al. 2019

Preprint

View full text Add to dashboard Cite

Non-Autoregressive Neural Machine Translation (NAT) achieves significant decoding speedup through generating target words independently and simultaneously. However, in the context of non-autoregressive translation, the word-level cross-entropy loss cannot model the target-side sequential dependency properly, leading to its weak correlation with the translation quality. As a result, NAT tends to generate influent translations with over-translation and under-translation errors. In this paper, we propose to train NAT to minimize the Bag-of-Ngrams (BoN) difference between the model output and the reference sentence. The bag-of-ngrams training objective is differentiable and can be efficiently calculated, which encourages NAT to capture the target-side sequential dependency and correlates well with the translation quality. We validate our approach on three translation tasks and show that our approach largely outperforms the NAT baseline by about 5.0 BLEU scores on WMT14 En↔De and about 2.5 BLEU scores on WMT16 En↔Ro.

show abstract

Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information

Cited by 3 publications

References 32 publications

Improving Non-autoregressive Generation with Mixup Training

Improving Non-autoregressive Generation with Mixup Training

Sequence-Level Training for Non-Autoregressive Neural Machine Translation

Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation

Contact Info

Product

Resources

About