Automated Item Generation: The Future of Medical Education Assessment

Royal, Kenneth D.; Hedgpeth, Mari‐Wells; Jeon, Tae Yeon; Colford, Cristin

doi:10.33590/emjinnov/10313113

Cited by 8 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While repeated iterations and parameter tweaking (e.g., by adjusting top_p and temperature to increase or decrease linguistic diversity) can often offer quick relief, some PIG executions may yield less useful items than others. Importantly, no form of automated item generation currently available fully obviates the need for formal quality checks and human review (Goldenfein, 2019; Köbis & Mossink, 2021; Royal et al, 2019). That is, even within the best iterations, most of what the PIG—or any other comparable algorithm—generates will not be usable and the onus is one the researcher to read, assess, and organize the extensive algorithmic output to select suitable items.…”

Section: Discussionmentioning

confidence: 99%

“…The aforementioned process also highlights other significant start-up costs that come with AIG: First, in the form described above, AIG is anything but automatic and extensive preparatory steps are required before any automation takes place (Kosh et al, 2019). Second, in order to actually use computer algorithms, one must either be able to train them—which requires expertise and large computational storing and processing capacities (Fetaya et al, 2020)—or gain access to existing software, which has hitherto not been widely available (Royal et al, 2019).…”

Section: Automated Item Generationmentioning

confidence: 99%

See 1 more Smart Citation

Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development.

Götz¹,

Maertens²,

Loomba³

et al. 2024

Psychological Methods

View full text Add to dashboard Cite

Measurement is at the heart of scientific research. As many-perhaps most-psychological constructs cannot be directly observed, there is a steady demand for reliable self-report scales to assess latent constructs. However, scale development is a tedious process that requires researchers to produce good items in large quantities. In this tutorial, we introduce, explain, and apply the Psychometric Item Generator (PIG), an opensource, free-to-use, self-sufficient natural language processing algorithm that produces large-scale, humanlike, customized text output within a few mouse clicks. The PIG is based on the GPT-2, a powerful generative language model, and runs on Google Colaboratory-an interactive virtual notebook environment that executes code on state-of-the-art virtual machines at no cost. Across two demonstrations and a preregistered fivepronged empirical validation with two Canadian samples (N Sample 1 = 501, N Sample 2 = 773), we show that the PIG is equally well-suited to generate large pools of face-valid items for novel constructs (i.e., wanderlust) and create parsimonious short scales of existing constructs (i.e., Big Five personality traits) that yield strong performances when tested in the wild and benchmarked against current gold standards for assessment. The PIG does not require any prior coding skills or access to computational resources and can easily be tailored to any desired context by simply switching out short linguistic prompts in a single line of code. In short, we present an effective, novel machine learning solution to an old psychological challenge. As such, the PIG will not require you to learn a new language-but instead, speak yours.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Automated Item Generationmentioning

confidence: 99%

Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development.

Götz¹,

Maertens²,

Loomba³

et al. 2024

Psychological Methods

View full text Add to dashboard Cite

show abstract

“…7 However, there is little work into generating large volumes of clinical vignettes in an automated way. While variations of a technique called Automated Item Generation (AIG) (a template-based, expert-informed process) 4 have been used in the past, we were unable to find examples in the literature of using chatbots or conversational AI for this purpose. This study explores the use of a modern conversational agent (aka chatbot), specifically ChatGPT, for generating and evaluating clinical vignettes.…”

Section: Introductionmentioning

confidence: 95%

“…3 Development of these health education items is also expensive, with a similarly constructed item, multiple choice test questions, estimated at a cost of $1500-2500 each. 4 Previous studies have proposed systematic frameworks for developing vignettes for health professionals, 2 methods for evaluating vignette content validity, 5 and integrating co-design into the vignette creation process. 6 A recent review that aimed to synthesize vignette methodology in qualitative healthcare research suggested that this methods-focused work was necessary, arguing that the variability between development methods demonstrated the need for a more rigorous, transparent vignette development process.…”

Section: Introductionmentioning

confidence: 99%

ChatGPT for Clinical Vignette Generation, Revision, and Evaluation

Benoit

2023

Preprint

View full text Add to dashboard Cite

Objective To determine the capabilities of ChatGPT for rapidly generating, rewriting, and evaluating (via diagnostic and triage accuracy) sets of clinical vignettes. Design We explored the capabilities of ChatGPT for generating and rewriting vignettes. First, we gave it natural language prompts to generate 10 new sets of 10 vignettes, each set for a different common childhood illness. Next, we had it generate 10 sets of 10 vignettes given a set of symptoms from which to draw. We then had it rewrite 15 existing pediatric vignettes at different levels of health literacy. Fourth, we asked it to generate 10 vignettes written as a parent, and rewrite these vignettes as a physician, then at a grade 8 reading level, before rewriting them from the original parent's perspective. Finally, we evaluated ChatGPT for diagnosis and triage for 45 clinical vignettes previously used for evaluating symptom checkers. Setting and participants ChatGPT, a publicly available, free chatbot. Main outcome measures Our main outcomes for de novo vignette generation were whether ChatGPT followed vignette creation instructions consistently, correctly, and listed reasonable symptoms for the disease being described. For generating vignettes from pre-existing symptom sets, we examined whether the symptom sets were used without introducing extra symptoms. Our main outcome for rewriting existing standardized vignettes to match patient demographics, and rewriting vignettes between styles, was whether symptoms were dropped or added outside the original vignette. Finally, our main outcomes examining diagnostic and triage accuracy on 45 standardized patient vignettes were whether the correct diagnosis was listed first, and if the correct triage recommendation was made. Results ChatGPT was able to quickly produce varied contexts and symptom profiles when writing vignettes based on an illness name, but overused some core disease symptoms. It was able to use given symptom lists as the basis for vignettes consistently, adding one additional (though appropriate) symptom from outside the list for one disease. Pediatric vignettes rewritten at different levels of health literacy showed more complex symptoms being dropped when writing at low health literacy in 87.5% of cases. While writing at high health literacy, it added a diagnosis to 80% of vignettes (91.7% correctly diagnosed). Symptoms were retained in 90% of cases when rewriting vignettes between viewpoints. When presented with 45 vignettes, ChatGPT identified illnesses with 75.6% (95% CI, 62.6% to 88.5%) first-pass diagnostic accuracy and 57.8% (95% CI, 42.9% to 72.7%) triage accuracy. Its use does require monitoring and has caveats, which we discuss. Conclusions ChatGPT was capable, with caveats and appropriate review, of generating, rewriting, and evaluating clinical vignettes.

show abstract

“…The most popular tool for testing in medical education are exams comprised of multiplechoice questions (MCQs) (Douthit et al, 2021;Grainger et al, 2018). Besides allowing for instant scoring, MCQs are suitable for achievement testing since they cover different skills and enable the assessment of numerous candidates in a cost and resource efficient manner (Royal et al, 2018;Rust et al, 2020).Writing MCQs is complex since experts must outline and replicate cognitive problem-solving skills (Billings et al, 2020;. In order to plan this procedure, experts must share their knowledge and collaborate behind the scenes.…”

Section: The Role Of Multiple-choice Questions In Medical Educationmentioning

confidence: 99%

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

Falcão

Pereira

Gonçalves³

et al. 2023

Adv in Health Sci Educ

View full text Add to dashboard Cite

Automatic Item Generation (AIG) refers to the process of using cognitive models to generate test items using computer modules. It is a new but rapidly evolving research area where cognitive and psychometric theory are combined into digital framework. However, assessment of the item quality, usability and validity of AIG relative to traditional item development methods lacks clarification. This paper takes a top-down strong theory approach to evaluate AIG in medical education. Two studies were conducted: Study I—participants with different levels of clinical knowledge and item writing experience developed medical test items both manually and through AIG. Both item types were compared in terms of quality and usability (efficiency and learnability); Study II—Automatically generated items were included in a summative exam in the content area of surgery. A psychometric analysis based on Item Response Theory inspected the validity and quality of the AIG-items. Items generated by AIG presented quality, evidences of validity and were adequate for testing student’s knowledge. The time spent developing the contents for item generation (cognitive models) and the number of items generated did not vary considering the participants' item writing experience or clinical knowledge. AIG produces numerous high-quality items in a fast, economical and easy to learn process, even for inexperienced and without clinical training item writers. Medical schools may benefit from a substantial improvement in cost-efficiency in developing test items by using AIG. Item writing flaws can be significantly reduced thanks to the application of AIG's models, thus generating test items capable of accurately gauging students' knowledge.

show abstract

Automated Item Generation: The Future of Medical Education Assessment

Cited by 8 publications

References 9 publications

Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development.

Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development.

ChatGPT for Clinical Vignette Generation, Revision, and Evaluation

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

Contact Info

Product

Resources

About