Using OpenAI GPT to Generate Reading Comprehension Items

Abstract

The purpose of this study is to introduce and evaluate a method for generating reading comprehension items using template‐based automatic item generation. To begin, we describe a new model for generating reading comprehension items called the text analysis cognitive model assessing inferential skills across different reading passages. Next, the text analysis cognitive model is used to generate reading comprehension items where examinees are required to read a passage and identify the irrelevant sentence. The sentences for the generated passages were created using OpenAI GPT‐3.5. Finally, the quality of the generated items was evaluated. The generated items were reviewed by three subject‐matter experts. The generated items were also administered to a sample of 1,607 Grade‐8 students. The correct options for the generated items produced a similar level of difficulty and yielded strong discrimination power while the incorrect options served as effective distractors. Implications of augmented intelligence for item development are discussed.

Using a hybrid of artificial intelligence and template-based method in automatic item generation to create multiple-choice questions in medical education: Hybrid AIG

“…Similar to our approach, a recent study incorporated a large language model into the process of developing reading comprehension items. 26 While addressing a critical issue in item development for a non-healthcare setting, its direct application to medical education is challenging due to the inherent complexities of health professions education. Furthermore, this approach integrates AI only into generating unique sentences based on rules imposed by experts, leaving the essential cognitive work dependent on expert input, which remains inefficient for medical education.…”

“…Therefore, our findings diverge from an important conclusion drawn in the study on the successful generation of reading comprehension items. 26 In that study, humans designed the cognitive model due to the necessity of testing with clear intentions to fulfill specific requirements and constraints. While we agree with the importance of clarity and constraints, we cannot fully subscribe to the interpretation that "it is neither possible nor desirable to create specifications and instructions using artificial intelligence" 26 .…”