Using Automated Procedures to Generate Test Items That Measure Junior High Science Achievement

Gierl, Mark J.; Latifi, Syed; Lai, Hollis; Matovinovic, Donna; Boughton, Keith A.

doi:10.4018/978-1-4666-9441-5.ch022

Cited by 5 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These outcomes highlight the benefit of using a strong theory approach: if cognitive features are identified and the items generated adequate for testing, item features that predict test performance may be specified and controlled . Consequently, these results encourage the use of of items generated using cognitive modeling processes in operational administrations (Gierl et al, 2016a(Gierl et al, , 2016b.…”

Section: Item Qualitymentioning

confidence: 60%

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

Falcão

Pereira

Gonçalves³

et al. 2023

Adv in Health Sci Educ

View full text Add to dashboard Cite

Automatic Item Generation (AIG) refers to the process of using cognitive models to generate test items using computer modules. It is a new but rapidly evolving research area where cognitive and psychometric theory are combined into digital framework. However, assessment of the item quality, usability and validity of AIG relative to traditional item development methods lacks clarification. This paper takes a top-down strong theory approach to evaluate AIG in medical education. Two studies were conducted: Study I—participants with different levels of clinical knowledge and item writing experience developed medical test items both manually and through AIG. Both item types were compared in terms of quality and usability (efficiency and learnability); Study II—Automatically generated items were included in a summative exam in the content area of surgery. A psychometric analysis based on Item Response Theory inspected the validity and quality of the AIG-items. Items generated by AIG presented quality, evidences of validity and were adequate for testing student’s knowledge. The time spent developing the contents for item generation (cognitive models) and the number of items generated did not vary considering the participants' item writing experience or clinical knowledge. AIG produces numerous high-quality items in a fast, economical and easy to learn process, even for inexperienced and without clinical training item writers. Medical schools may benefit from a substantial improvement in cost-efficiency in developing test items by using AIG. Item writing flaws can be significantly reduced thanks to the application of AIG's models, thus generating test items capable of accurately gauging students' knowledge.

show abstract

Section: Item Qualitymentioning

confidence: 60%

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

Falcão

Pereira

Gonçalves³

et al. 2023

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…Items that are rejected have no value because they fail to meet the content and item development standards. As a result, these two categories of items are unacceptable (Gierl et al., 2016).…”

Section: Methodsmentioning

confidence: 99%

“…The key characteristic of a logical structures cognitive model is that the content for the item can vary, but the idea, formula, algorithm, and/or logical outcome required to manipulate the content is fixed. Cognitive model development using logical structures is well-documented for generating items in content areas such as science (Gierl & Lai, 2017;Gierl et al, 2016) and mathematics (Gierl & Lai, 2016b;Gierl et al, 2015). For example, a logic structure model can be used to solve word problems that measure interval and ratio (problems and scenarios).…”

Section: Cognitive Models For Aigmentioning

confidence: 99%

Using OpenAI GPT to Generate Reading Comprehension Items

Sayin,

Gierl

2024

Educational Measurement

Self Cite

View full text Add to dashboard Cite

The purpose of this study is to introduce and evaluate a method for generating reading comprehension items using template‐based automatic item generation. To begin, we describe a new model for generating reading comprehension items called the text analysis cognitive model assessing inferential skills across different reading passages. Next, the text analysis cognitive model is used to generate reading comprehension items where examinees are required to read a passage and identify the irrelevant sentence. The sentences for the generated passages were created using OpenAI GPT‐3.5. Finally, the quality of the generated items was evaluated. The generated items were reviewed by three subject‐matter experts. The generated items were also administered to a sample of 1,607 Grade‐8 students. The correct options for the generated items produced a similar level of difficulty and yielded strong discrimination power while the incorrect options served as effective distractors. Implications of augmented intelligence for item development are discussed.

show abstract

“…Unfortunately, there are few empirical methods available for quantifying the similarity of generated test items, [36] and hence to date, similarity is often established more subjectively using judgments from test development specialists. To address this limitation in literature, we describe two measure of item similarity that can be used to evaluate the comparability of the generated items.…”

Section: Discussionmentioning

confidence: 99%

“…Given the high cost of item development, the proposed empirical methods for reviewing and identifying commonality within large item banks will help focus resources on unique items rather than on item editing and revision. [36] Nevertheless, the semantic space constructed in this study uses a corpora from a relatively small item bank. We expect that if the semantic space had been constructed using a large sample of operational test items, we would have access to more co-occurrence information thereby leading to the construction of word vectors with close to true semantic meanings.…”

Section: Discussionmentioning

confidence: 99%

Information-based methods for evaluating the semantics of automatically generated test items

Latifi

Gierl

Ren

et al. 2016

AIR

Self Cite

View full text Add to dashboard Cite

Multiple-choice questions are the popular type of test items that are used for testing the knowledge of health-science students in north America and elsewhere. The motivation of this article is to present the recent advances in the automatic item generation (AIG) and to propose a novel unsupervised approach that extends the information-based Compositional Distributional Semantic Model (CDSM) to measure the semantic relatedness among the pool of automatically generated items. We have used operational item bank from the medical science domain for developing the CDSM and demonstrated our approach using the concepts from AIG research. We illustrated our approach using eleven item models from the medical education domain, and discussed the possible applications to advance the AIG research.

show abstract

Using Automated Procedures to Generate Test Items That Measure Junior High Science Achievement

Cited by 5 publications

References 13 publications

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation

Using OpenAI GPT to Generate Reading Comprehension Items

Information-based methods for evaluating the semantics of automatically generated test items

Contact Info

Product

Resources

About