Stopping Rules for Computer Adaptive Testing When Item Banks Have Nonuniform Information

Abstract

The standard error (SE) stopping rule, which terminates a computer adaptive test (CAT) when the SE is less than a threshold, is effective when there are informative questions for all trait levels. However, in domains such as patient reported outcomes, the items in a bank might all target one end of the trait continuum (e.g., negative symptoms), and the bank may lack depth for many individuals. In such cases, the predicted standard error reduction (PSER) stopping rule will stop the CAT even if the SE threshold has not been reached, and can avoid administering excessive questions that provide little additional information. By tuning the parameters of the PSER algorithm, a practitioner can specify a desired tradeoff between accuracy and efficiency. Using simulated data for the PROMIS Anxiety and Physical Function banks, we demonstrate that these parameters can substantially impact CAT performance. When the parameters were optimally tuned, the PSER stopping rule was found to outperform the SE stopping rule overall and particularly for individuals not targeted by the bank, and presented roughly the same number of items across the trait continuum. Therefore, the PSER stopping rule provides an effective method for balancing the precision and efficiency of a CAT.

Psychometrics of patient-reported outcomes measurement information system in von Willebrand disease, inherited platelet function disorders, and rare bleeding disorders

“…Patients with no problems or disease symptoms have to answer a maximum of 12 questions to reach the CAT stopping rule. Adjustments to this stopping rule should be evaluated to explore if it is possible to reduce questionnaire burden [ 61 ]. Second, other aspects of feasibility such as the comprehensibility of the PROMIS CATs and SF-36v2 in patients with low health literacy and the cost for administration should be evaluated.…”

Measurement and structural invariance of a neuropsychological battery among Middle Eastern/North African, Black, and White older adults.

“…For each individual, a difference was considered meaningful if the absolute value of this difference exceeded 0.3 (Fyffe et al, 2011; Gibbons et al, 2017). This commonly used precision threshold comes from the standard error stopping rule in the computer adaptive testing literature (Morris et al, 2020), which corresponds to a reliability of .9 (Nunnally & Bernstein, 1995). If greater than 5% of the overall sample exhibited meaningful differences, then it was concluded that the identified measurement noninvariance was salient and should not be ignored in any group comparisons.…”

Active Sampling of Multiple Sources for Sequential Estimation

“…This enforces a hard constraint on its stopping time. Even though some investigations introduce an adaptive stopping time [45][46][47][48], the stopping time may not exceed the number of items in the pool. In contrast, we consider an infinite-horizon setting, in which each experiment can be sampled as many times as necessary.…”