RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Abstract

Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrievalaugmented or prompting-based critique generators. 1

Self-powered Vs. high speed ZnO-based photodetectors

“…CRITIC (Gou et al, 2023) proposes the use of a suite of specialized tools for a variety of tasks such as code interpreters, calculators, or search engines to generate critics for the LLM's generated output. Moreover, approaches such as REFINER (Paul et al, 2023), CodeRL and RL4F (Akyurek et al, 2023) propose to train a specialized critic to provide feedback to the generator model.…”

“…Various methods have been proposed to tackle this problem (Pan et al, 2023). From training-time correction Li et al, 2019;Jauregi Unanue et al, 2021;Zelikman et al, 2022;Huang et al, 2022) to post output generation refinement (Madaan et al, 2023;Shinn et al, 2023;Zhang et al, 2023;Pan et al, 2023;Yu et al, 2023;Gou et al, 2023;Paul et al, 2023;Akyurek et al, 2023), these methods have shown the impact that iterative self-refinement and proper feedback can have on the performance of LLMs.…”