Automatic Generation of Figural Analogies With the IMak Package

Abstract

Automatic Item Generation (AIG) techniques are offering innovative ways to produce test items as they overcome many disadvantages involving standard item writing, such as time-consuming work and resource-intensive demands. Although this field is relatively new, it is progressing at a high speed, and several contributions have been accomplished. Nevertheless, a scarce amount of AIG software evidencing favorable psychometric properties of the generated items has been made accessible to the broad scientific community. This research had two goals: first, to present an empirical study of items produced with the aid of the Item Maker (IMak) package available online and, second, to present IMak itself for the automatic generation of figural analogies. We were particularly interested in assessing whether automatically created figural analogy rules could predict item psychometric difficulty. A total of 23 items were generated and administered to 307 participants, 49.51% from Germany. The mean age was 28.61 (SD = 10.19) and 57.65% of the participants were female. Results reveal adequate psychometric properties including convergent validity, that most of the manipulated rules contribute to item difficulty, and that rule-based difficulty prediction is possible to some extent. In other words, psychometric quality of the generated items is supported, which reveals the utility of the IMak package in assessment settings. Finally, the package is presented and its functions for figural analogy item generation are further described.

Automatic item generation: foundations and machine learning-based approaches for assessments

“…In this review, commonly used input types for item generation can be divided into two groups: (a) structured inputs, such as item model templates (e.g., Gierl et al, , 2016Latifi et al, 2013;Colvin et al, 2016;Attali, 2018;Blum and Holling, 2018), and (b) unstructured inputs such as available written material (e.g., Khodeir et al, 2018;von Davier, 2019;Zhou and Huang, 2019;Attali et al, 2022). A third input type can be described as a combination of both structured and unstructured inputs (e.g., Atapattu et al, 2012;.…”

“…In the educational context, item models are commonly used (Gierl et al, 2008;Gierl and Lai, 2013;Blum and Holling, 2018;Embretson and Kingston, 2018;Pugh et al, 2020). An item model is defined as "… a template that specifies the features in an item that can be manipulated" (LaDuca et al, 1986;Bejar et al, 2003).…”

Development of a Public-Domain Measure of Two-Dimensional Rotation Ability and Preliminary Evidence for Discriminant Validity among Occupations

“…IRT scores were obtained based on the final set of 2D rotation ability items using the "mirt" package (Chalmers 2012). IRT scores were also generated for the other cognitive abilities in the ICAR framework and for the SPI traits, based on calibrations from previously published works (Mather et al, forthcoming;Blum et al 2016;Blum and Holling 2018;Condon and Revelle 2014;Gühne et al 2021). Sum scoring was used to score the big five personality traits, which were assessed using the same items as the SPI traits and represent a broader level of the trait hierarchy.…”

Feasibility assurance: a review of automatic item generation in medical assessment

“…In step one, specialist identify the exam content and outline a framework with the necessary knowledge and skills to formulate a diagnosis, forming a cognitive model structure (Blum & Holling, 2018 ; Pugh et al, 2016 ). Within the model, a problem specific to the test item is identified, along with different scenarios and the information needed to solve it.…”