Toward Diverse Text Generation with Inverse Reinforcement Learning

Abstract

Text generation is a crucial task in NLP. Recently, several adversarial generative models have been proposed to improve the exposure bias problem in text generation. Though these models gain great success, they still suffer from the problems of reward sparsity and mode collapse. In order to address these two problems, in this paper, we employ inverse reinforcement learning (IRL) for text generation. Specifically, the IRL framework learns a reward function on training data, and then an optimal policy to maximum the expected total reward. Similar to the adversarial models, the reward and policy function in IRL are optimized alternately. Our method has two advantages: (1) the reward function can produce more dense reward signals.(2) the generation policy, trained by "entropy regularized" policy gradient, encourages to generate more diversified texts. Experiment results demonstrate that our proposed method can generate higher quality texts than the previous methods.

Generating Sentiment-Preserving Fake Online Reviews Using Neural Language Models and Their Human- and Machine-Based Detection

“…This may be the reason that the countermeasures could easily detect fake reviews. To generate more robust reviews, we plan to develop a method that generates reviews with more diversity [35]. We also plan to develop a countermeasure for detecting these generated reviews.…”

ARAML: A Stable Adversarial Training Framework for Text Generation

“…As for the discriminator, RankGAN (Lin et al, 2017) replaced traditional discriminator with a ranker to learn the relative ranking information between the real texts and generated ones. Inverse reinforcement learning (Shi et al, 2018) used a trainable reward approximator as the discriminator to provide dense reward signals at each generation step. DPGAN ) introduced a language model based discriminator and regarded cross-entropy as rewards to promote the diversity of generation results.…”

“…MaliGAN: A variant of SeqGAN that optimizes the generator with a normalized maximum likelihood objective (Che et al, 2017). IRL: This inverse reinforcement learning method replaces the discriminator with a reward approximator to provide dense rewards (Shi et al, 2018). RAML: A RL approach to incorporate MLE objective into RL training framework, which regards BLEU as rewards (Norouzi et al, 2016).…”

“…Although widely used, MLE suffers from the exposure bias problem (Bengio et al, 2015;Ranzato et al, 2016): during test, the model sequentially predicts the next word conditioned on its previous generated words while during training conditioned on ground-truth words. To tackle this problem, generative adversarial networks (GAN) with reinforcement learning (RL) training approaches have been introduced to text generation tasks Che et al, 2017;Lin et al, 2017;Shi et al, 2018;, where the discriminator is trained to distinguish real and generated text samples to provide reward signals for the generator, and the generator is optimized via policy gradient .…”

Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning

“…This method enables us to both make use of an efficient adversarial formulation and recover a more precise reward function for open-domain dialogue training. Unlike Shi et al (2018), we design a specific reward function structure to measure the reward of each word in generated sentences while taking account of the dialogue context. We also consider two human evaluation settings to assess the overall performance of our model.…”