Conducting simulation studies for computerized adaptive testing using SimulCAT: an instructional piece

Abstract

Computerized adaptive testing (CAT) technology is widely used in a variety of licensing and certification examinations administered to health professionals in the United States. Many more countries worldwide are expected to adopt CAT for their national licensing examinations for health professionals due to its reduced test time and more accurate estimation of a test-taker’s performance ability. Continuous improvements to CAT algorithms promote the stability and reliability of the results of such examinations. For this reason, conducting simulation studies is a critically important component of evaluating the design of CAT programs and their implementation. This report introduces the principles of SimulCAT, a software program developed for conducting CAT simulation studies. The key evaluation criteria for CAT simulation studies are explained and some guidelines are offered for practitioners and test developers. A step-by-step tutorial example of a SimulCAT run is also presented. The SimulCAT program supports most of the methods used for the 3 key components of item selection in CAT: the item selection criterion, item exposure control, and content balancing. Methods for determining the test length (fixed or variable) and score estimation algorithms are also covered. The simulation studies presented include output files for the response string, item use, standard error of estimation, Newton-Raphson iteration information, theta estimation, the full response matrix, and the true standard error of estimation. In CAT simulations, one condition cannot be generalized to another; therefore, it is recommended that practitioners perform CAT simulation studies in each stage of CAT development.

Evaluating a Computerized Adaptive Testing Version of a Cognitive Ability Test Using a Simulation Study

“…The SEE is estimated as shown belowwhere TIF is the test information function for the specific test form that each test-taker completed. It should also be noted that SEE statistic is often used as a termination criterion in variable-length CAT systems (Han, 2018a).…”

“…An efficient means to evaluate a CAT system is via a Monte Carlo (MC) simulation technique. An MC configuration provides several useful information by simulating both item and person parameters, including person true θ values, true item parameters (usually a and b ), which could also be used to evaluate important aspects of a newly developed psychometric tool (applications can include tests of model fit, differential item functioning, test performance, etc (Han, 2018a). Simulation studies are also very helpful because they comprise the only applied means to test and evaluate a CAT system before its implementation with real participants.…”

Exploring Counselor‐Client Agreement on Clients’ Work Capacity in Established and Consultative Dyads

“…CAT is a computer‐based testing of scale items that adapts to the examinee's ability level. The computational simulations of assessments allow for flexibility and efficiency in administration (Han, 2018). Item content was conceptually grounded in the World Health Organization (2018) International Classification of Functioning, Disability, and Health and was developed using a rigorous process that included extensive literature review, cognitive testing, and focus group input from individuals with work disabilities as well as from work disability experts.…”

Development of a Computerized Adaptive Testing for Internet Addiction

“…These abilities were the estimated values of 1,000 simulated persons. In addition, the CAT-IA performance was evaluated via several statistical indices, including conditional bias (CBIAS), conditional mean absolute error (CMAE), conditional root mean square error (CRMSE), and conditional standard error of estimation (CSEE) across all θ areas (Han, 2018). Simulation study 1 was done in the R package catR (Version 3.12; Magis and Barrada, 2017).…”