The Implications of Large Language Models for CS Teachers and Students

Abstract

Identifying and resolving logic errors can be one of the most frustrating challenges for novices programmers. Unlike syntax errors, for which a compiler or interpreter can issue a message, logic errors can be subtle. In certain conditions, buggy code may even exhibit correct behavior -in other cases, the issue might be about how a problem statement has been interpreted. Such errors can be hard to spot when reading the code, and they can also at times be missed by automated tests. There is great educational potential in automatically detecting logic errors, especially when paired with suitable feedback for novices. Large language models (LLMs) have recently demonstrated surprising performance for a range of computing tasks, including generating and explaining code. These capabilities are closely linked to code syntax, which aligns with the next token prediction behavior of LLMs. On the other hand, logic errors relate to the runtime performance of code and thus may not be as well suited to analysis by LLMs. To explore this, we investigate the performance of two popular LLMs, GPT-3 and GPT-4, for detecting and providing a novice-friendly explanation of logic errors. We compare LLM performance with a large cohort of introductory computing students (𝑛 = 964) solving the same error detection task. Through a mixed-methods analysis of student and model responses, we observe significant improvement in logic error identification between the previous and current generation of LLMs, and find that both LLM generations significantly outperform students.We outline how such models could be integrated into computing education tools, and discuss their potential for supporting students when learning programming.

Prompt Problems: A New Programming Exercise for the Generative AI Era

“…Recently, the focus has started to shift from assessing the capabilities of LLMs to using them in teaching and learning practice [20]. For example, Sarsa et al showed that LLMs can generate viable programming exercises including test cases and explanations [27], and Liffiton et al describe the use of an LLM-powered teaching assistant with guardrails suitable for computing courses [18].…”

ChatGPT to Support Teachers - Classification, Analysis and Examples of Applications

“…Weitere wissenschaftliche Untersuchungen äußern Bedenken hinsichtlich der Vermittlung von Didaktik durch ChatGPT, wobei Aspekte wie Praktikabilität, Ethik, Transparenz, Datenschutz, Gleichheit und Wohlergehen thematisiert werden (Yan 2023). Ein spezifischer inhaltlicher Fokus liegt dabei auf der automatisierten Generierung von Aufgaben, wie beispielsweise Multiple-Choice-Fragen (Bitew et al 2022) oder Programmierübungen (Sarsa et al 2022;MacNeil et al 2023). Innerhalb dieser Diskussionen wird betont, dass trotz der automatisierten Erstellung die Qualität der generierten Inhalte einer ständigen Kontrolle durch Fachkräfte unterliegen muss (Sarsa et al 2022).…”