Reinforcement Learning and Feedback Control

Abstract

In recent years, large language models (LLMs) have had a dramatic impact on various sub-fields of AI, most notably on natural language understanding tasks. However, there is widespread agreement that the logical reasoning capabilities of contemporary LLMs are, at best fragmentary (i.e., may work well on some problem instances but fail dramatically on others). While traditional LLM finetuning approaches (e.g., those that use human feedback) do address this problem to some degree, they suffer from many issues, including unsound black-box reward models, difficulties in collecting preference data, and sparse scalar reward values.To address these challenges, we propose a new training/fine-tuning paradigm we refer to as Reinforcement Learning via Symbolic Feedback (RLSF), which is aimed at enhancing the reasoning capabilities of LLMs. In the RLSF setting, the LLM that is being trained/fine-tuned is considered as the RL agent, while the environment is allowed access to reasoning or domain knowledge tools (e.g., solvers, algebra systems). Crucially, in RLSF, these reasoning tools can provide feedback to the LLMs via poly-sized certificates (e.g., proofs), that characterize errors in the LLM-generated object with respect to some correctness specification. The ability of RLSF-based training/fine-tuning to leverage certificate-generating symbolic tools enables sound fine-grained (token-level) reward signals to LLMs, and thus addresses the limitations of traditional reward models mentioned above.Via extensive evaluations, we show that our RLSF-based fine-tuning of LLMs outperforms traditional approaches on two different applications, namely, program synthesis from natural language pseudo-code to programming language (C++), and solving the Game of 24. For the former, RLSF-tuned LLMs show a significant improvement (e.g., +52.64% in compilation accuracy and +31.43% in functional correctness for Google's CodeGemma-2b) compared to supervised fine-tuning. Further, CodeGemma-2b gets +34.82% in compilation accuracy and +17.01% in functional correctness compared to ChatGPT (100× larger). Similarly, for the Game of 24, we observed +25% success rate using Meta's Llama2-7b compared to traditional methods, and +7% success rate compared to ChatGPT (25× larger). A takeaway is that fine-tuning via RLSF enables relatively smaller LLMs to significantly outperform orders of magnitude larger models (e.g., ChatGPT).Preprint. Under review.

Infusing behavior science into large language models for activity coaching

“…In reality, it is important to consistently align the coach responses to user need throughout a conversation and adapt as the dialogue unfolds. Methods such as reinforcement learning with human feedback can help to offer this adaptability (29). Finally, the behaviour model used was a simplistic one that conceptualizes user behaviour only along three axes -future studies could consider using more sophisticated behavior science frameworks, which may help to better target coach actions.…”

Machine Learning Approaches in Bioengineering for Biosignal Processing

“…Unsupervised learning methods use unlabeled data to discover hidden patterns or structures in the data, such as grouping similar EEG signals into clusters or reducing the dimensionality of highdimensional biosignals [44]. Reinforcement learning methods use feedback from the environment to learn an optimal policy or action for a given state, such as controlling a prosthetic limb using EMG signals [45]. For each category, we will introduce some of the most popular and effective methods and their advantages and limitations, as well as some case studies and applications in biosignal processing.…”

Infusing behavior science into large language models for activity coaching

“…In reality, it is important to consistently align the coach responses to user need throughout a conversation and adapt as the dialogue unfolds. Methods such as reinforcement learning with human feedback can help to offer this adaptability [15]. Finally, the behaviour model used was a simplistic one that conceptualizes user behaviour only along three axesfuture studies could consider using more sophisticated behavior science frameworks, which may help to better target coach actions.…”

“…over-specialization of the model [12]. Knowledge infusion is an active area of research and many other methods exist including customizing training objectives [13], reinforcement learning with human feedback [14,15], in-context learning via prompt engineering or priming [16,17] and many associated prompt design variants [18][19][20][21]. There have also been numerous strategies to ensemble knowledge infusion techniques, including post-hoc re-ranking or summarization of model outputs to further align the model with the task of interest [22,23].…”