Three Sources of Validation Evidence Needed to Evaluate the Quality of Generated Test Items for Medical Licensure

Gierl, Mark J.; Swygert, Kimberly A.; Matovinovic, Donna; Kulesher, Allison; Lai, Hollis

doi:10.1080/10401334.2022.2119569

Cited by 4 publications

(14 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, the quality and the psychometric properties of generated items to multiple-choice testing are mostly undocumented (Gierl et al, 2016;Shappell et al, 2021). Generated items need to be of a high caliber for AIG to be truly useful (Gierl et al, 2022b). The few studies that have compared manually and AIG written items generally support the use of the latter.…”

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

“…While technology makes it easier to administer test contents to students, it also raises questions regarding test security and validity (Patel, 2021).Validation processes are key to ensuring the validity of assessment tools and consists of assembling evidence to provide a scientific basis for the interpretation of scores (American Educational Research Association, 2018). Item development is essential to validity assessments because it records the procedures and results required to produce high-quality test content (Gierl et al, 2022b). A fundamental issue for AIG research regards the validation of its processes (Gierl et al, 2022b;Shin, 2021).…”

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

“…Item development is essential to validity assessments because it records the procedures and results required to produce high-quality test content (Gierl et al, 2022b). A fundamental issue for AIG research regards the validation of its processes (Gierl et al, 2022b;Shin, 2021). To date, few research has been devoted to collecting evidence to support the validity of AIG (Falcão et al, 2022;Gierl et al, 2022b;Rafatbakhsh et al, 2020).…”

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

“…A fundamental issue for AIG research regards the validation of its processes (Gierl et al, 2022b;Shin, 2021). To date, few research has been devoted to collecting evidence to support the validity of AIG (Falcão et al, 2022;Gierl et al, 2022b;Rafatbakhsh et al, 2020). Since incorporating cognitive models into test design and development is required to support validity arguments for test-based inferences, we have reason to suppose that AIG incorporates validity evidence in its methods (Gierl et al, 2022b;Leighton & Gierl, 2011).…”

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

“…To date, few research has been devoted to collecting evidence to support the validity of AIG (Falcão et al, 2022;Gierl et al, 2022b;Rafatbakhsh et al, 2020). Since incorporating cognitive models into test design and development is required to support validity arguments for test-based inferences, we have reason to suppose that AIG incorporates validity evidence in its methods (Gierl et al, 2022b;Leighton & Gierl, 2011). Therefore, by obtaining evidences of validity, we would be taking a critical step to support greater use of AIG in educational assessment.…”

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

See 4 more Smart Citations

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

Falcão

Pereira

Gonçalves³

et al. 2023

Adv in Health Sci Educ

View full text Add to dashboard Cite

Automatic Item Generation (AIG) refers to the process of using cognitive models to generate test items using computer modules. It is a new but rapidly evolving research area where cognitive and psychometric theory are combined into digital framework. However, assessment of the item quality, usability and validity of AIG relative to traditional item development methods lacks clarification. This paper takes a top-down strong theory approach to evaluate AIG in medical education. Two studies were conducted: Study I—participants with different levels of clinical knowledge and item writing experience developed medical test items both manually and through AIG. Both item types were compared in terms of quality and usability (efficiency and learnability); Study II—Automatically generated items were included in a summative exam in the content area of surgery. A psychometric analysis based on Item Response Theory inspected the validity and quality of the AIG-items. Items generated by AIG presented quality, evidences of validity and were adequate for testing student’s knowledge. The time spent developing the contents for item generation (cognitive models) and the number of items generated did not vary considering the participants' item writing experience or clinical knowledge. AIG produces numerous high-quality items in a fast, economical and easy to learn process, even for inexperienced and without clinical training item writers. Medical schools may benefit from a substantial improvement in cost-efficiency in developing test items by using AIG. Item writing flaws can be significantly reduced thanks to the application of AIG's models, thus generating test items capable of accurately gauging students' knowledge.

show abstract

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

Section: Aig Versus Manual Item Writingmentioning

confidence: 99%

See 3 more Smart Citations

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

Falcão

Pereira

Gonçalves³

et al. 2023

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

Exploring quality criteria and evaluation methods in automated question generation: A comprehensive survey

Gorgun,

Bulut

2024

Educ Inf Technol

View full text Add to dashboard Cite

A Workflow for Minimizing Errors in Template‐Based Automated Item‐Generation Development

2024

Educational Measurement

View full text Add to dashboard Cite

The template‐based automated item‐generation (TAIG) approach that involves template creation, item generation, item selection, field‐testing, and evaluation has more steps than the traditional item development method. Consequentially, there is more margin for error in this process, and any template errors can be cascaded to the generated items. Therefore, it is essential to eliminate the source of errors and ensure the quality of the template so items can be problem‐free. The article introduces a process to reduce template errors at the early stage of template development, minimize the impact of template errors on generated items, and increase the survival rates of generated items. The article also discusses a statistical method to establish confidence in the quality of the template by systematically examining the quality of the generated items. The proposed method can reduce the review process for some items generated from a template.

show abstract

Three Sources of Validation Evidence Needed to Evaluate the Quality of Generated Test Items for Medical Licensure

Cited by 4 publications

References 17 publications

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

Exploring quality criteria and evaluation methods in automated question generation: A comprehensive survey

A Workflow for Minimizing Errors in Template‐Based Automated Item‐Generation Development

Contact Info

Product

Resources

About