Machine learning and logic: a new frontier in artificial intelligence

Seshia, Sanjit A.; Jha, Somesh

doi:10.1007/s10703-023-00430-1

Cited by 1 publication

(2 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite their impressive performance, these models often struggle with tasks that require complex reasoning [7,8,9]. This limitation has sparked a growing interest in exploring methods that can enhance the reasoning capabilities of LLMs, particularly through the incorporation of corrective feedback loops between learning and reasoning processes [10,11]. For example, Kambhampati et al [11] state that "LLMs cannot plan themselves but can play a variety of constructive roles in solving planning tasks-especially as approximate knowledge sources and candidate plan generators in the so-called LLM-Modulo Frameworks in conjunction with external sound model-based verifiers.…”

Section: Introductionmentioning

confidence: 99%

“…The concept of incorporating reasoning tools into machine learning in general, and LLMs in particular, is rooted in the idea of combining the strengths of these two sub-fields of AI [12,10,11,13]. While LLMs excel at capturing statistical patterns and generating fluent text, they can fail to perform sound reasoning and generate text that is logically coherent.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning and Feedback Control

Hua¹

2021

Reinforcement Learning Aided Performance Optimization of Feedback Control Systems

View full text Add to dashboard Cite

In recent years, large language models (LLMs) have had a dramatic impact on various sub-fields of AI, most notably on natural language understanding tasks. However, there is widespread agreement that the logical reasoning capabilities of contemporary LLMs are, at best fragmentary (i.e., may work well on some problem instances but fail dramatically on others). While traditional LLM finetuning approaches (e.g., those that use human feedback) do address this problem to some degree, they suffer from many issues, including unsound black-box reward models, difficulties in collecting preference data, and sparse scalar reward values.To address these challenges, we propose a new training/fine-tuning paradigm we refer to as Reinforcement Learning via Symbolic Feedback (RLSF), which is aimed at enhancing the reasoning capabilities of LLMs. In the RLSF setting, the LLM that is being trained/fine-tuned is considered as the RL agent, while the environment is allowed access to reasoning or domain knowledge tools (e.g., solvers, algebra systems). Crucially, in RLSF, these reasoning tools can provide feedback to the LLMs via poly-sized certificates (e.g., proofs), that characterize errors in the LLM-generated object with respect to some correctness specification. The ability of RLSF-based training/fine-tuning to leverage certificate-generating symbolic tools enables sound fine-grained (token-level) reward signals to LLMs, and thus addresses the limitations of traditional reward models mentioned above.Via extensive evaluations, we show that our RLSF-based fine-tuning of LLMs outperforms traditional approaches on two different applications, namely, program synthesis from natural language pseudo-code to programming language (C++), and solving the Game of 24. For the former, RLSF-tuned LLMs show a significant improvement (e.g., +52.64% in compilation accuracy and +31.43% in functional correctness for Google's CodeGemma-2b) compared to supervised fine-tuning. Further, CodeGemma-2b gets +34.82% in compilation accuracy and +17.01% in functional correctness compared to ChatGPT (100× larger). Similarly, for the Game of 24, we observed +25% success rate using Meta's Llama2-7b compared to traditional methods, and +7% success rate compared to ChatGPT (25× larger). A takeaway is that fine-tuning via RLSF enables relatively smaller LLMs to significantly outperform orders of magnitude larger models (e.g., ChatGPT).Preprint. Under review.

show abstract

Section: Introductionmentioning

confidence: 99%