Latent-Class-Based Item Selection for Computerized Adaptive Progress Tests

Abstract

Standard computerized adaptive testing (CAT) methods require an underlying item response theory (IRT) model. An item bank can be constructed from the IRT model, and subsequent items can be selected with maximum information at the examinee's estimated ability level. IRT models, however, do not always fit test data exactly. In such situations, it is not possible to employ standard CAT methods without violating assumptions. To extend the scope of adaptive testing, this research shows how latent class analysis (LCA) can be used in item bank construction. In addition, the research investigates suitable item selection algorithms using KullbackLeibler (KL) information for item banks based on LCA. The KL information values can be used to select items and to construct an adaptive test. Simulations show that item selection based on KL information outperformed random selection of items in progress testing. The effectiveness of the selection algorithm is evaluated, and a possible scoring for the new adaptive item selection with two classes is proposed. The applicability of the methods is illustrated by constructing a computerized adaptive progress test (CAPT) on an example data set drawn from Misfit in IRT ModelsIn some situations, however, the standard IRT framework that underlies CAT does not fit the data, for example, during the calibration phase when building a CAT. Global fit of the IRT models to an item bank can be tested by looking at global fit measures such as the Q1 test, R1 test, and/or likelihood ratio tests (Andersen, 1973;Suárez-Falcón & Glas, 2003). These testing methods for global model fit can be first pointers to misfit. Concerns raised by global indications of misfit can be caused by violations of one or more assumptions of the IRT model. Item response functions can be flat instead of S-shaped, and problems with assumed local independence and unidimensionality can occur (Yang & Kao, 2014).If many items from the item bank show misfit, constructing an item bank might be impossible or only possible with heavy violations of the assumptions of standard IRT models. In these cases, other models can be applied to the data, such as multidimensional IRT models or latent variable mixture models. Multidimensional IRT can be modeled when multiple constructs in the data disturb the model fit (Béguin & Glas, 2001). Latent variable mixture modeling is useful in populations with heterogenous rather than homogenous samples (Sawatzky, Ratner, Kopec, Wu, & Zumbo, 2016). Sometimes, however, these other IRT models do not fit Research on LCA and CATThe present research appears to be the first to propose LCA in combination with CAT in an actual testing situation. Macready and Dayton (1992) investigated the use of LCA in CAT,concluding that the combination of CAT with LCA allows for conceptually simpler models than CAT with IRT. Macready and Dayton also acknowledged that LCA has fewer untestable assumptions. Their goal was to classify respondents into the appropriate latent class with CAT and to obtain an acceptable level of...

Computerized Adaptive Testing Without IRT for Flexible Measurement and Prediction

“…Hence, in FlexCAT the calibration stage consists of finding an estimate of p with a model of choice. Besides IRT models, candidate models for estimating p include the latent class model (LCM; e.g., Vermunt et al, 2008;Linzer, 2011;Van Buuren & Eggen, 2017), the divisive LCM (Van der Palm et al, 2016), kernel estimation methods (e.g., Li & Racine, 2003), and decision trees (e.g., Ho, 1995;Yan et al, 2004).…”

Development of an Assessment Model for Electric Circuit Courses Based on "Free Campus Learning (MBKM) According to Industry Needs" Using an Expert System

“…The majority of them continue to believe that assessment is performed only for students (Ambiyar et al, 2019) and applied exclusively to online learning activities (Ramadania & Aswadi, 2020;Sherly et al, 2021). Improvements and additions are made which include a measurement instrument for learning activities by creating an assessment model for electrical circuit course based on the MBKM program using an expert system, in the form of a computer-based adaptive test, as has been done in research (Nalova & Shalanyuy, 2017;Winarno, 2012;Li et al, 2015;Boeve et al, 2015;Liu et al, 2016;Van Buuren et al, 2017). The objective of the study is Development of an Assessment Model for Electric Circuit Courses Based on "Free Campus Learning (MBKM) according to industry needs" Using an Expert System, with the urgency of the research stemmed from the difficulty of lecturers in understanding and developing learning assessment models.…”

Assessment of ICT in Tertiary Education Applying Structural Equation Modeling and Rasch Model

“…Its emphases have covered, but have not been limited to improvement in the quality of teaching, learning, research, and the pursuit of research productivity. Contemporary psychologists have conducted numerous assessment investigations that apply computer-based analytical tools such as structural equation modeling (SEM; Gibbons et al, 2018;Lee & Vlack, 2018;Scherer et al, 2017;Zhu et al, 2018), hierarchical linear modeling (Areepattamannil & Khine, 2017), the Rasch model (Campbell & Bond, 2017;Mok et al, 2015), partial least squares (Onn et al, 2018), and computerized adaptive testing (Buuren & Eggen, 2017;Huebner et al, 2018). However, despite a decade of practice, the literature suggests that researchers or psychometricians may need to better comprehend and apply these analytical tools to improve the quality of practice measurements in higher education.…”