A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond

Abstract

Non-autoregressive (NAR) generation, which is first proposed in neural machine translation (NMT) to speed up inference, has attracted much attention in both machine learning and natural language processing communities. While NAR generation can significantly accelerate inference speed for machine translation, the speedup comes at the cost of sacrificed translation accuracy compared to its counterpart, auto-regressive (AR) generation. In recent years, many new models and algorithms have been designed/proposed to bridge the accuracy gap between NAR generation and AR generation. In this paper, we conduct a systematic survey with comparisons and discussions of various non-autoregressive translation (NAT) models from different aspects. Specifically, we categorize the efforts of NAT into several groups, including data manipulation, modeling methods, training criterion, decoding algorithms, and the benefit from pre-trained models. Furthermore, we briefly review other applications of NAR models beyond machine translation, such as dialogue generation, text summarization, grammar error correction, semantic parsing, speech synthesis, and automatic speech recognition. In addition, we also discuss potential directions for future exploration, including releasing the dependency of KD, dynamic length prediction, pre-training for NAR, and wider applications, etc. We hope this survey can help researchers capture the latest progress in NAR generation, inspire the design of advanced NAR models and algorithms, and enable industry practitioners to choose appropriate solutions for their applications. The web page of this survey is at https://github.com/LitterBrother-Xiao/Overview-of-Non-autoregressive-Applications.

ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing

“…However, due to the autoregressive nature, searching for the word at current position only considers the information from the left side, not the full context. Besides, the autoregressive nature tends to bring mode collapse problem [74], resulting in captions with less diversity. Moreover, the timecost of iterative gradient-update is high, especially for long captions.…”

“…However, such autoregressive generation often results in issues such as sequential error accumulation and lack of diversity [13,74]. Further, for zero-shot IC, the sequential searching-order is lack of flexible.…”

InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation

“…Non-autoregressive (NAR) generation refers to a method of generating sequences where each element is generated independently, without relying on previously generated elements, allowing for faster parallel generation but potentially sacrificing the generation accuracy (Xiao et al, 2022). Recently, diffusion models have demonstrated powerful generative capabilities in image generation tasks, gradually becoming a new paradigm in generative models.…”