Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits

Abstract

We present SECOND THOUGHTS, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, SECOND THOUGHTS not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness. * Work done during the internship at Dartmouth College. 2 We take the log-probability predicted by the LM, log Pr(y|x), which is the conditional log-probability of generating option y given input context x. We then compute its exponential for better readability. Such a protocol is also adopted by BIG-Bench: https://github.com/google/BIG-bench.36th Conference on Neural Information Processing Systems (NeurIPS 2022).the source to produce the target (Figure 2 (b)). This way the model learns how to recover from a value-unaligned, poisoned context during the generation phase. Augmented Edits ModelingDP-based Edits Inference. Given two text strings, source and target, one can find unlimited ways to edit source to produce target. Thus, we apply two constraints onto the editing: (1) the edits should be combinations of generic editing operations-inserting, deleting, and replacing a single token; (2) each edit operation has a cost and our goal is to infer the chain-of-edits that has minimum cost. Under these constraints, the edits inference problem can be converted to a token-level "edit distance problem" (Jurafsky, 2000), which can be solved by dynamic programming (DP). We modify the algorithm to be able to receive customized editing costs (e.g., insert-1, delete-1, replace-2), to try to model different preferences on editing. We use special tokens to mark the start/end of editing and the new content to be inserted/replaced, and develop a decipher module that can translate the edit operations produced by DP into natural language (see §A.1 for a visualization of the whole process, and §A.3 for more discussion on edit based models).Augmented Edits Modeling (AEM). To augment the edits, we run the DP algorithm on the same source and target pairs with a variety of editing costs 4 to create a collection of chain-of-edits for each source-target pair, which we call positive demonstrations (y + ). We then fine-tune an LM on these source-edits-target text inputs (recall that the edits are turned into natural language). We call this Augmented Edits Modeling (AEM). Different from common language modeling, AEM includes the labor-free decomposition (i.e., the editing steps) into the training object, whereas prior works either train on costly manually-created decomposition (Ouyang et al., 2022; or, rather than training, prompt with such decomposition Nye et al., 2021). We also construct negative demonstrations (y − ) by us...

Using ChatGPT for human–computer interaction research: a primer

“…The inclusion of colourblind or otherwise impaired road users in traffic indeed represents an important human value, and perhaps also one that a human researcher would have wanted to consider; however, strictly speaking, it was not something that was prominently featured in the 30 summaries and therefore did not emerge in the human content analysis. The challenge of alignment, that is, to what extent the output that large language models generate aligns with human values, is an ongoing research topic [ 80 , 81 ]. It can also be noted here that these values may be context-dependent, whereby in some cases, a researcher might want to receive a mechanistic output, such for a meta-analysis, and in other cases, they would prefer an output that takes into account more general human values such as safety and inclusiveness.…”

Recent Advances towards Safe, Responsible, and Moral Dialogue Systems: A Survey

“…Some researchers found that only a small proportion of the whole response (e.g., one or two words) needs to be fixed. Thus, An edition module takes effect after the generation to fix some problems in some works [75,78]. Similarly, text style transfer or rephrasing from toxicity to non-toxicity can also be plugged in this stage [26,66].…”

The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values

“…The first group generally seeks value alignment, i.e., some notion of steering language models towards producing societally-desirable text Liu et al, 2021a). We note a variety of vague goals such as to reduce "non-normative" (Peng et al, 2020) or "immoral" text (Liu et al, 2023c); to generate more "pro-social" or "legitimate" text (Bakker et al, 2022); or to encourage that LLM technologies have a "positive impact on society" (Liu et al, 2023b). Specific motivations include minimising toxic or offensive language (Dinan et al, 2019;Xu et al, 2021a;Ju et al, 2022;Scheurer et al, 2022;Korbak et al, 2023); improving safety (Liu et al, 2021a;Thoppilan et al, 2022;Ganguli et al, 2022;Jin et al, 2022); adapting to ethical or moral scenarios (Forbes et al, 2020;Jin et al, 2022); or achieving political ideological balance (Liu et al, 2021b).…”

“…Explicit comparisons collected on model outputs are used to reveal the preferences of human raters (Gao et al, 2018;Ziegler et al, 2019;Askell et al, 2021;Jaques et al, 2020;Stiennon et al, 2020;Ganguli et al, 2022;Glaese et al, 2022). 6 More finegrained feedback includes binary or Likert scale questions on text attributes (Nakano et al, 2021;Menick et al, 2022;Thoppilan et al, 2022); natural language comments (Ju et al, 2022;Scheurer et al, 2022); or edits (Hancock et al, 2019;Liu et al, 2023c). Ideal demonstrations are used to ground norm-dependent or ethical judgements (Forbes et al, 2020;Pyatkin et al, 2022;Jin et al, 2022), or in combination with ratings to prime model behaviour (Nakano et al, 2021;Wu et al, 2021;Ouyang et al, 2022;Bakker et al, 2022).…”

“…(Step 4): Fine-tune a RL policy (another LM) that generates text autoregressively, whilst the PM provides a reward signal. Often, the policy is updated using the PPO algorithm (Ziegler et al, 2019;Stiennon et al, 2020;Nakano et al, 2021) and a KL-penalty term is applied to control deviations from the base model (Jaques et al, 2019;Ziegler et al, 2019;Stiennon et al, 2020;Nakano et al, 2021;Menick et al, 2022;Ouyang et al, 2022;Liu et al, 2023c). This pipeline can be implemented in offline, online or batched settings (see Ziegler et al, 2019).…”