Using Large Language Models to Enhance Programming Error Messages

Leinonen, Juho; Hellas, Arto; Sarsa, Sami; Reeves, Brent; Denny, Paul; Prather, James; Becker, Brett A.

doi:10.1145/3545945.3569770

Cited by 97 publications

(37 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There has also been extensive work on improving the programming-error-messages by designing customized environments [9,10]. As discussed earlier, a recent study used Codex to enhance these error messages [6]; however, our work is different as we focus on generating high-precision feedback with a tuneable precision knob.…”

Section: Related Workmentioning

confidence: 99%

“…P b along the following binary attributes: (i) P f is syntactically correct and is obtained by making a small number of edits to fix P b ; (ii) X is complete, i.e., contains adequate information about all errors and required fixes; (iii) X is correct, i.e., the provided information correctly explains errors and required fixes; (iv) X is comprehensible, i.e., easy to understand, presented in a readable format, and doesn't contain redundant information. These attributes are inspired by evaluation rubrics used in literature [6,[25][26][27]. In our evaluation, feedback quality is evaluated via ratings by experts along these four attributes.…”

Section: Preliminariesmentioning

confidence: 99%

“…Given P b and P f as inputs, PyFiXV's Stage-2 generates a natural language explanation X describing errors/fixes. We use Codex-Complete in this stage as it is naturally suited to generate text by completing a prompt [1,5,6]. A crucial ingredient of Stage-2 is the annotated dataset D shot used to select few-shot examples when querying Codex-Complete (see Figure 4).…”

Section: Stage-2: Generating Explanationmentioning

confidence: 99%

“…students in a large introductory programming course [3]. Subsequently, recent works have shown promising results in using Codex on various programming education scenarios, including generating new programming assignments [4], providing code explanations [5], and enhancing programming-error-messages [6].…”

Section: Introductionmentioning

confidence: 99%

“…High precision is imperative in building educators' trust before deploying such an AI-driven technology in classrooms. A recent work investigated enhancing the default programming-error-messages using Codex [6]; one of the takeaways, quoted from their paper, is "The key implications of this work are that program- The student forgets to concatenate the string literal with the variable. We can fix the error by adding a plus sign between the string literal and the variable.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

Phung¹,

Cambronero²,

Gulwani³

et al. 2023

Preprint

View full text Add to dashboard Cite

Large language models trained on code (LLMCs), such as Codex, hold great promise in enhancing programming education by automatically generating feedback for students. We investigate using LLMCs to generate feedback for fixing syntax errors in Python programs, a key scenario in introductory programming. More concretely, given a student's buggy program, our goal is to generate feedback comprising a fixed program along with a natural language explanation describing the errors/fixes, inspired by how a human tutor would give feedback. While using LLMCs is promising, the critical challenge is to ensure high precision in the generated feedback, which is imperative before deploying such technology in classrooms. The main research question we study is: Can we develop LLMCs-based feedback generation techniques with a tunable precision parameter, giving educators quality control over the feedback that students receive? To this end, we introduce PyFiXV, our technique to generate high-precision feedback powered by Codex. The key idea behind PyFiXV is to use a novel run-time validation mechanism to decide whether the generated feedback is suitable for sharing with the student; notably, this validation mechanism also provides a precision knob to educators. We perform an extensive evaluation using two real-world datasets of Python programs with syntax errors and show the efficacy of PyFiXV in generating high-precision feedback.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Preliminariesmentioning

confidence: 99%

Section: Stage-2: Generating Explanationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

Phung¹,

Cambronero²,

Gulwani³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

What Is a Person? Emerging Interpretations of AI Authorship and Attribution

Moulaison‐Sandy

2023

Proceedings of the Association for Information Science and Tech

View full text Add to dashboard Cite

Recently, the scholarly community has been eagerly exploring how AI‐produced content should be integrated into both academic writing and scholarly publishing. This paper investigates the prevailing responses to the introduction of ChatGPT in November 2022 and the interest that has been afforded it by both the academy and the publishing industry. A review of the published literature on aspects of ChatGPT authorship was carried out, finding that government and the publishing industry have unequivocally asserted that large language models (LLMs) like ChatGPT do not possess the traits of a person and are not able to author texts as a result. Other approaches, including practice, have been less vehement. To assess the integration of instructions on referencing ChatGPT using APA, top Google hits in the .edu domain were collected and analyzed over a 6‐week period from March 14 to April 18, 2023, a time during which official recommendations of the APA Style were finalized. Findings reveal that librarians were quick to provide guidance, but slow to update that guidance, contributing to the potential for misunderstanding the affordances of and best practices for work with LLMs.

show abstract

Investigating the Role of ChatGPT in Supporting Text-Based Programming Education for Students and Teachers

Wieser,

Schöffmann,

Stefanics

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Teaching text-based programming poses significant challenges in both school and university contexts. This study explores the potential of ChatGPT as a sustainable didactic tool to support students, freshmen, and teachers. By focusing on a beginner’s course with examples also relevant to vocational schools, we investigated three research questions. First, the extent to which ChatGPT assists students in solving and understanding initial examples; secondly, the feasibility of teachers utilizing the chatbot for grading student solutions; and finally, the additional support ChatGPT provides in terms of teaching. Our findings demonstrate that ChatGPT offers valuable guidance for teachers in terms of assessment and grading and aids students in understanding and optimizing their solutions.

show abstract

Using Large Language Models to Enhance Programming Error Messages

Cited by 97 publications

References 26 publications

Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models

What Is a Person? Emerging Interpretations of AI Authorship and Attribution

Investigating the Role of ChatGPT in Supporting Text-Based Programming Education for Students and Teachers

Contact Info

Product

Resources

About