How understanding large language models can inform the use of ChatGPT in physics education

Abstract

The paper aims to fulfil three main functions: (1) to serve as an introduction for the physics education community to the functioning of Large Language Models (LLMs), (2) to present a series of illustrative examples demonstrating how prompt-engineering techniques can impact LLMs performance on conceptual physics tasks and (3) to discuss potential implications of the understanding of LLMs and prompt engineering for physics teaching and learning. We first summarise existing research on the performance of a popular LLM-based chatbot (ChatGPT) on physics tasks. We then give a basic account of how LLMs work, illustrate essential features of their functioning, and discuss their strengths and limitations. Equipped with this knowledge, we discuss some challenges with generating useful output with ChatGPT-4 in the context of introductory physics, paying special attention to conceptual questions and problems. We then provide a condensed overview of relevant literature on prompt engineering and demonstrate through illustrative examples how selected prompt-engineering techniques can be employed to improve ChatGPT-4’s output on conceptual introductory physics problems. Qualitatively studying these examples provides additional insights into ChatGPT’s functioning and its utility in physics problem solving. Finally, we consider how insights from the paper can inform the use of LMMs in the teaching and learning of physics.

ChatGPT as a tool for honing teachers’ Socratic dialogue skills

“…The first relevant finding of this case study is that we were able to find an introductory conceptual physics question that ChatGPT-4 did not answer correctly. This further supports previous findings, suggesting that its performance on even basic physics tasks significantly limits the usefulness of the chatbot as a physics tutor [15].…”

“…This does not seem to be the case in physics, where even the state-of-the-art chatbot ChatGPT-4 3 [14] still has some way to go before its performance can be considered expert-like. Surprisingly, this is true not only for advanced physics topics but even for introductory conceptual physics, where it can still clumsily fail [15]. Therefore, the effectiveness of the technology is contingent upon the users' expertise in the subject matter, their ability to craft useful prompts, and critically evaluate the outputs.…”

Assessment of Students Use of Generative Artificial Intelligence: Prompting Strategies and Prompt Engineering in Chemistry Education

“…"Set the Scene" improves the answer due to the context sensitivity of large language models. 44 "Be specific" is similar to what Zheng et al propose as "Detailed Instruction". 46 "Simplify your language" is sometimes contradicting specificity, but a consequence of how large language models work with natural language.…”

“…46 "Simplify your language" is sometimes contradicting specificity, but a consequence of how large language models work with natural language. 44 "Structure the output" again is a principle found with Zheng et al 46 "Share Feedback" is a further strategy developed due to the high dependency of large language models on the data provided to it. While this prompting framework is sometimes contradictory and, in some ways, fuzzy, it might serve as a facilitator for reflecting on students' own prompts.…”

“…There is an extensive overview of such strategies, including how and why they work. 44 However, there is little knowledge about what strategies students use in their generative AI interactions. Also, critical evaluation of the generated output is extremely important to overcome inaccuracies and hallucinations.…”