Deep Reinforcement Learning for Sequence-to-Sequence Models

Abstract

In recent times, sequence-to-sequence (seq2seq) models have gained a lot of popularity and provide state-ofthe-art performance in a wide variety of tasks such as machine translation, headline generation, text summarization, speech to text conversion, and image caption generation. The underlying framework for all these models is usually a deep neural network comprising an encoder and a decoder. Although simple encoderdecoder models produce competitive results, many researchers have proposed additional improvements over these seq2seq models, e.g., using an attention-based model over the input, pointergeneration models, and self-attention models. However, such seq2seq models suffer from two common problems: 1) exposure bias and 2) inconsistency between train/test measurement. Recently, a completely novel point of view has emerged in addressing these two problems in seq2seq models, leveraging methods from reinforcement learning (RL). In this survey, we consider seq2seq problems from the RL point of view and provide a formulation combining the power of RL methods in decision-making with seq2seq models that enable remembering long-term memories. We present some of the most recent frameworks that combine concepts from RL and deep neural networks. Our work aims to provide insights into some of the problems that inherently arise with current approaches and how we can address them with better RL models. We also provide the source code for implementing most of the RL models discussed in this paper to support the complex task of abstractive text summarization and provide some targeted experiments for these RL models, both in terms of performance and training time.

Generalization in Generation: A closer look at Exposure Bias

“…Shortcomings of maximum-likelihood training for sequence generation have often been discussed (Ding and Soricut, 2017;Leblond et al, 2018;Ranzato et al, 2016), but without pointing to generalization as the key aspect. An overview of recent deep reinforcement learning methods for conditional generation can be found in (Keneshloo et al, 2018). Our proposed approach follows work by Ding et al (2017) and Tan et al (2018) by employing both, policy and reward for exploration.…”

This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation

“…For the Pointer-Generator Network from See et al (2017), we follow their implementation 4 and use a batch size 16. For Paulus et al (2018), we use an implementation from Keneshloo et al (2018) 5 . We did not include the intra-temporal attention and the intra-decoder attention because they hurt the performance.…”

A Survey of Automatic Generation of Source Code Comments: Algorithms and Techniques

“…Other information retrieval based algorithms [10], [55] Single-encoder based algorithms [31], [33], [41], [89] Multiple-encoder based algorithms [32], [80] Other algorithms [64], [68], [81] Note that in the process of collecting papers, we first performed two types of searches for related papers: (1) Online library search for papers containing keywords including "code + comment", "comment", "code + summary" and "summary" in the fields of title, abstract and index terms of the papers from ACM Digital Library, IEEE Xplore Digital Library, DBLP, Google Scholar and arXiv.org. (2) Specific search of major conference proceedings and journals in software engineering and artificial intelligence, including IEEE ICSE, IEEE FSE, IEEE/ACM ASE, IEEE TSE, ACM TOSEM, EMSE, AAAI and IJCAI.…”

“…That is, they use two encoders in the classical encoder-decoder framework. Under this framework, they exploit reinforcement learning model to solve two issues: exposure bias and inconsistency between train and test measurement [81]. They leverage an actor network and a critic network to jointly determine the next best word at each time step.…”