Human-centric dialog training via offline reinforcement learning

Abstract

How can we train a dialog model to produce better conversations by learning from human feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback from real-time, open-ended conversations, which we then use to train and improve the models using offline reinforcement learning (RL). We identify implicit conversational cues including language similarity, elicitation of laughter, sentiment, and more, which indicate positive human feedback, and embed these in multiple reward functions. A wellknown challenge is that learning an RL policy in an offline setting usually fails due to the lack of ability to explore and the tendency to make over-optimistic estimates of future reward. These problems become even harder when using RL for language models, which can easily have a 20,000 action vocabulary and many possible reward functions. We solve the challenge by developing a novel class of offline RL algorithms. These algorithms use KL-control to penalize divergence from a pretrained prior language model, and use a new strategy to make the algorithm pessimistic, instead of optimistic, in the face of uncertainty. We test the resulting dialog model with ratings from 80 users in an open-domain setting and find it achieves significant improvements over existing deep offline RL approaches. The novel offline RL method is viable for improving any existing generative dialog model using a static dataset of human feedback.

Text Generation with Efficient (Soft) Q-Learning

“…The key advantage of off-policy updates is that samples from other sources, such as human-written text, can be used, making them more data efficient than on-policy methods. Previous work has used either importance weighted PG [42,68,25] or Q-learning based algorithms [16,23,38]. However, the off-policy methods have been considered to be less stable.…”

“…For example, the Q-learning performance relies heavily on how accurate the learned Q-function assesses the quality of intermediate subsequences -a challenging task due to the sparse reward signals (e.g., reward is received only after the whole sequence is generated). Further, previous work has largely focused on the extreme of using only off-policy data, mostly for offline training of chatbots [23]. As a result, the opportunity of directly improving the reward (as in on-policy updates) for other rich tasks is missed.…”

Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior

“…Explicit human feedback has also been incorporated into reinforcement learning methods (Knox and Stone, 2009;Pilarski et al, 2011;Daniel et al, 2015;Mathewson and Pilarski, 2016;Warnell et al, 2018;MacGlashan et al, 2017;Arumugam et al, 2019), including in the context of dialogue system learning (Liu et al, 2018). Jaques et al (2020) study forming a reward from implicit feedback for non-task-oriented dialogue language generation, by training multiple models to detect linguistic signals, such as sentiment and lexical overlap, that correlate with explicit user feedback. Learning from users has also been studied by asking users to rank system outputs (e.g., Wilson et al, 2012;Christiano et al, 2017), including for instruction following (Wang et al, 2016) and summarization (Stiennon et al, 2020).…”

Offline Reinforcement Learning with Soft Behavior Regularization

“…Prior works tackle this problem by ensuring that the learned policy stays "close" to the behavior policy via behavior regularization. This is achieved either by explicit constraints on the learned policy to only select actions where (s, a) has sufficient support under the behavior distribution (Fujimoto, Meger, and Precup 2019;Ghasemipour, Schuurmans, and Gu 2021); or by adding a regularization term that calculates some divergence metrics between the learned policy and the behavior policy (Wu, Tucker, and Nachum 2019;Siegel et al 2019;Zhang, Kuppannagari, and Prasanna 2020;Dadashi et al 2021), e.g., KL divergence (Jaques et al 2020) or Maximum Mean Discrepancy (MMD) (Kumar et al 2019). While straightforward, these methods lack guaranteed performance improvement against the behavior policy.…”