The Role of Item Models in Automatic Item Generation

Gierl, Mark J.; Lai, Hollis

doi:10.1080/15305058.2011.635830

Cited by 40 publications

(28 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These developments can be summarized as improvements that occur prior to, during, or after item generation. Research focused on improvements that occur prior to item generation address design‐related issues such as cognitive model development (e.g., Embretson & Yang, ; Gierl & Lai, ; Gierl, Lai, & Turner, ), item model development (e.g., Gierl, Zhou, & Alves, ; Gierl & Lai, ), and test space designs for AIG (e.g., Bejar et al., ; Embretson & Yang, ; Huff, Alves, Pellegrino, & Kaliski, ; Lai & Gierl, ; Luecht, ). Research focused on improvements that occur during item generation include many of the technological solutions for AIG (e.g., Gierl et al., ; Gütl, Lankmayr, Weinhofer, & Höfler, ; Higgins, ; Higgins, Futagi, & Deane, ; Mortimer, Stroulia, & Yazdchi, ), including the use of language‐based approaches for item generation that draw on natural language processing and rule‐based artificial intelligence (e.g., Gütl et al., ), frame‐semantic representations (e.g., Deane & Sheehan, 2003, unpublished data; Higgins et al., ), and schema theory (e.g., Singley & Bennett, ).…”

mentioning

confidence: 99%

“…Research focused on improvements that occur after generation focus on item precalibration methods and the development of statistical models for generated items (e.g., Embretson, ; Geerlings, Glas, & van der Linden, ; Glas & van der Linden, ; Sinharay & Johnson, , ; Sinharay, Johnson, & Williamson, ). Because of these important research developments, AIG has been used to create millions of new items in diverse content areas, including but not limited to K‐12 levels in subjects such as Language Arts, Social Studies, Science, Mathematics (Gierl et al., ; Gierl & Lai, , ), and advanced placement (AP) Biology (Alves, Gierl, & Lai, ); in psychological domains such as spatial (Bejar, ), abstract (Embretson, ), figural inductive (Arendasy, ), and quantitative reasoning (Arendasy & Sommer, ; Embretson & Daniels, ; Sinharay & Johnson, ), as well as word fluency (Arendasy, Sommer, & Mayr, ), visual short‐term memory (Hornke, ), and mental rotation (Arendasy & Sommer, ); and in licensure and certification testing areas such as nursing, architecture, and medicine (Wendt, Kao, Gorham, & Woo, ; Gierl et al., ; Gierl, Lai, & Turner, ).…”

mentioning

confidence: 99%

See 1 more Smart Citation

Instructional Topics in Educational Measurement (ITEMS) Module: Using Automated Processes to Generate Test Items

Gierl

Lai

2013

Educational Measurement

Self Cite

View full text Add to dashboard Cite

Changes to the design and development of our educational assessments are resulting in the unprecedented demand for a large and continuous supply of content‐specific test items. One way to address this growing demand is with automatic item generation (AIG). AIG is the process of using item models to generate test items with the aid of computer technology. The purpose of this module is to describe and illustrate a template‐based method for generating test items. We outline a three‐step approach where test development specialists first create an item model. An item model is like a mould or rendering that highlights the features in an assessment task that must be manipulated to produce new items. Next, the content used for item generation is identified and structured. Finally, features in the item model are systematically manipulated with computer‐based algorithms to generate new items. Using this template‐based approach, hundreds or even thousands of new items can be generated with a single item model.

show abstract

mentioning

confidence: 99%

mentioning

confidence: 99%

Instructional Topics in Educational Measurement (ITEMS) Module: Using Automated Processes to Generate Test Items

Gierl

Lai

2013

Educational Measurement

Self Cite

View full text Add to dashboard Cite

show abstract

“…That is, AIG treats the item model as the unit of analysis where a single model is used to generate many items compared with a traditional item writing approach where the item is treated as the unit of analysis. The shift in the unit of analysis means that the SME manipulates only a small number of elements in each item model rather than manipulating every element in each new item (Gierl & Lai, ). This model‐based approach, when combined with the review process presented in our article, can yield large numbers of high‐quality items in an efficient and cost‐effective manner.…”

Section: Discussionmentioning

confidence: 99%

A Process for Reviewing and Evaluating Generated Test Items

Gierl

Lai

2016

Educational Measurement

Self Cite

View full text Add to dashboard Cite

Testing organization needs large numbers of high‐quality items due to the proliferation of alternative test administration methods and modern test designs. But the current demand for items far exceeds the supply. Test items, as they are currently written, evoke a process that is both time‐consuming and expensive because each item is written, edited, and reviewed by a subject‐matter expert. One promising approach that may address this challenge is with automatic item generation. Automatic item generation combines cognitive and psychometric modeling practices to guide the production of items that are generated with the aid of computer technology. The purpose of this study is to describe and illustrate a process that can be used to review and evaluate the quality of the generated item by focusing on the content and logic specified within the item generation procedure. We illustrate our process using an item development example from mathematics drawn from the Common Core State Standards and from surgical education drawn from the health sciences domain.

show abstract

“…Estos pasos se detallarán a continuación, utilizando un lenguaje similar al pseudocódigo para la creación de un programa de computadora, es decir, como si la intención real fuera manipular artificialmente los radicales (Irvine, 2002) de un Ítem-Modelo (Gierl & Lai, 2012;Lai, Alves, & Gierl, 2009) en un proceso de Generación Automática de Ítems (Arendasy, 2002;Freund, Hofer, & Holling, 2008). Los pasos presuponen que un ítem debe contener entre 2 y 8 opciones de respuesta, ya que cantidades mayores a 8 crean ítems con una estructura demasiado compleja y es posible que dicha complejidad distorsione las respuestas.…”

Section: Pautas Específicas Para La Construcción De íTemsunclassified

“…Es importante mencionar que existe una demanda creciente de miles de nuevos ítems para incluir en evaluaciones computarizadas por parte de agencias de evaluación muy conocidas (Gierl & Lai, 2012). Por ende, a medida que el desarrollo de ítems se complejiza, la demanda de cantidad y calidad de ítems excede con creces la capacidad de aquellos redactores que los arman del modo usual (Lai, Alves, & Gierl, 2009).…”

Section: Conclusionesunclassified

Modelización de ítems de matrices figurales y pautas específicas propuestas para su construcción

et al. 2015

View full text Add to dashboard Cite

Resumen: Este artículo da a conocer los resultados de la modelización de un Test de Analogías Figurales (TAF) utilizando los aportes conjuntos de la Teoría Clásica de Tests (TCT) y de la Teoría de Respuesta al Ítem (TRI), así como la posterior interpretación de las desventajas y posibles soluciones del diseño de dicho test. Se desarrolla el marco conceptual que dio origen al armado del TAF, así como los estudios realizados con esta prueba al administrarla a dos muestras. Una de las muestras corresponde a un estudio piloto con estudiantes de Psicología, mientras que la otra muestra consistió en estudiantes de carreras artísticas y de diseño. Los resultados principales revelan indicadores favorables de unidimensionalidad y confiabilidad, así como parámetros aceptables de discriminación (a), dificultad (b) y pseudoazar (c). Se discuten las desventajas del diseño de ítems con tres reglas y nuevas especificaciones en cuanto al armado de reactivos matriciales de 2x2. Palabras clave: Matrices; figuras; ítems; analogía; reglas.Title: Modelling figural matrix items and specification of guidelines for their construction. Abstract: This manuscript publicizes the results that concern the modelling of a Test of Figural Analogies (TFA) using Classical Test Theory (CTT) and Item Response Theory (IRT) on a joint basis, as well as the interpretation of the disadvantages and possible solutions of the test design. The theoretical background for the TFA construction is outlined, and studies accomplished after its administration to two samples is explained. One of these samples corresponds to a pilot study with Psychology students, while the other sample consisted of Art and Design college students. Main results indicate good unidimensionality and reliability, as well as acceptable discrimination (a), difficulty (b) and guessing value (c) parameters. Design disadvantages of three-rule based items are discussed, and new specifications for the 2x2 matrix item construction are given. El RA es un razonamiento no deductivo y, como tal, no asegura una conclusión verdadera cuando todas sus premisas son verdaderas. Esto quiere decir que es un razonamiento que no está basado en una conclusión necesaria, sino más bien probable. Por este motivo, se trata de un razonamiento ampliatorio, que permite razonar sobre lo posible y no sobre lo deductivamente válido. A diferencia de otros razonamientos no deductivos, como la inducción y la abducción, en el RA se establecen inferencias hacia casos particulares partiendo de premisas sobre otros casos particulares, de manera que nunca se parte de una generalidad ni tampoco se arriba a una generalidad de casos. Esto quiere decir que el RA va de lo particular a lo particular.La inferencia se realiza sobre un caso nuevo, poco conocido. El RA consiste en atribuirle a dicho caso propiedades que pertenecen a otros casos, los cuales son mejor conocidos que el anterior y de los que se establecen las premisas. En palabras de expertos como Gick y Holyoak (1980), Holyoak y Koh (1987) y Gentner (1983), el dominio nu...

show abstract

The Role of Item Models in Automatic Item Generation

Cited by 40 publications

References 19 publications

Instructional Topics in Educational Measurement (ITEMS) Module: Using Automated Processes to Generate Test Items

Instructional Topics in Educational Measurement (ITEMS) Module: Using Automated Processes to Generate Test Items

A Process for Reviewing and Evaluating Generated Test Items

Modelización de ítems de matrices figurales y pautas específicas propuestas para su construcción

Contact Info

Product

Resources

About