Improving mathematics assessment readability: Do large language models help?

Abstract

Background: Readability metrics provide us with an objective and efficient way to assess the quality of educational texts. We can use the readability measures for finding assessment items that are difficult to read for a given grade level. Hard-to-read math word problems can put some students at a disadvantage if they are behind in their literacy learning. Despite their math abilities, these students can perform poorly on difficult-to-read word problems because of their poor reading skills. Less readable math tests can create equity issues for students who are relatively new to the language of assessment. Less readable test items can also affect the assessment's construct validity by partially measuring reading comprehension.Objectives: This study shows how large language models help us improve the readability of math assessment items. Methods:We analysed 250 test items from grades 3 to 5 of EngageNY, an opensource curriculum. We used the GPT-3 AI system to simplify the text of these math word problems. We used text prompts and the few-shot learning method for the simplification task.Results and Conclusions: On average, GPT-3 AI produced output passages that showed improvements in readability metrics, but the outputs had a large amount of noise and were often unrelated to the input. We used thresholds over text similarity metrics and changes in readability measures to filter out the noise. We found meaningful simplifications that can be given to item authors as suggestions for improvement.Takeaways: GPT-3 AI is capable of simplifying hard-to-read math word problems.The model generates noisy simplifications using text prompts or few-shot learning methods. The noise can be filtered using text similarity and readability measures. The meaningful simplifications AI produces are sound but not ready to be used as a direct replacement for the original items. To improve test quality, simplifications can be suggested to item authors at the time of digital question authoring.

Assessing the Proficiency of Large Language Models in Automatic Feedback Generation: An Evaluation Study

“…The results of RQ1 showed that the GPT models were able to generate more readable feedback with greater consistency than human instructors. This is not surprising because the ability of LLMs to provide readable responses in various contexts has been demonstrated in the literature Li and Xing (2021); Patel et al (2023). For example, Li and Xing (2021) indicated that GPT-2 could provide readable and coherent replies to human learners' posts on MOOC discussion forums to a similar extent compared to human instructors.…”

Artificial intelligence on the advance to enhance educational assessment: Scientific clickbait or genuine gamechanger?

“…Patel et al (2023) contributed a paper that was directly aimed at teachers, too. Focusing on the output and action level, they used the generative large language model GPT‐3 to reduce the linguistic difficulty of mathematical word problems.…”

Learning Analytics in the Era of Large Language Models

“…In the era of LLMs, language model recommender systems have been proposed to increase transparency and control for students by enabling them to interact with the learning system using natural language [110]. LLMs can interpret natural language user profiles and use them to modulate learning materials for each session [111]. For example, Zhang et al [112] proposed a language model recommender system leveraging several language models, including GPT2 and BERT.…”