Validation of Computerized Adaptive Testing in an Outpatient Nonacademic Setting: The VOCATIONS Trial

Abstract

Objective Computerized adaptive tests (CAT) provide an alternative to fixed-length assessments for diagnostic screening and severity measurement of psychiatric disorders. We sought to cross-sectionally validate a suite of computerized adaptive tests for mental health (CAT-MH) in a community psychiatric sample. Methods 145 adult psychiatric outpatients and controls were prospectively evaluated with CAT for depression, mania and anxiety symptoms, compared to gold-standard psychiatric assessments including: Structured Clinical Interview for DSM IV-TR (SCID), Hamilton Rating Scale for Depression (HAM-D25), Patient Health Questionnaire (PHQ-9), Center for Epidemiologic Studies Depression Scale (CES-D), and Global Assessment of Functioning (GAF). Results Sensitivity and specificity for the computerized adaptive diagnostic test for depression (CAD-MDD) were .96 and .64, respectively (.96 and 1.00 for major depression versus controls). CAT for depression severity (CAT-DI) correlated well to standard depression scales HAM-D25 (r=.79), PHQ-9 (r=.90), CES-D (r=.90) and had OR=27.88 for current SCID major depressive disorder diagnosis across its range. CAT for anxiety severity (CAT-ANX) correlated to HAM-D25 (r=.73), PHQ-9 (r=.78), CES-D (r=.81), and had OR=11.52 for current SCID generalized anxiety disorder diagnosis across its range. CAT for mania severity (CAT-MANIA) did not correlate well to HAM-D25 (r=.31), PHQ-9 (r=.37), CES-D (r=.39), but had an OR=11.56 for a current SCID bipolar diagnosis across its range. Participants found the CAT-MH suite of tests acceptable and easy to use, averaging 51.7 items and 9.4 minutes to complete the full battery. Conclusions Compared to current gold-standard diagnostic and assessment measures, CAT-MH provides an effective, rapidly-administered assessment of psychiatric symptoms.

Depression in Emergency Department Patients and Association With Health Care Utilization

“…A prior study showed that the CAD‐MDD was on average shorter than the PHQ‐9 (an average of four items versus nine items) and that overall sensitivity and specificity for the CAD‐MDD was 0.95 and 0.87, respectively, compared to 0.70 and 0.91 for the PHQ‐9, compared to the Structured Clinical Interview for DSM‐IV (SCID) criterion standard . An independent validation study including patients presenting to an outpatient mental health clinic and healthy controls showed similar test performance . We have also recently validated the CAD‐MDD within our institution's primary care population with comparable results …”

Computerized Adaptive Diagnosis and Testing of Mental Health Disorders

“…Example adaptive testing session results are presented in Table 1. Results for the CAT inventories for anxiety (CAT-ANX) (Gibbons et al 2014) and mania (CAT-MANIA) (Achtyes et al 2015) closely paralleled those for the CAT-DI. Using an average of 12 adaptively administered items, we found correlations of r = 0.94 and r = 0.92 for the total anxiety and mania item bank scores.…”

“…Application of unidimensional models to multidimensional data can result in biased trait estimates (e.g., severity), underestimates of uncertainty (Gibbons et al 2007), and exclusion of large numbers of informative items from the bank. We have developed the underlying statistical theory and methodology necessary to apply multidimensional CAT to the measurement of depression, anxiety, and mania/hypomania symptom severity (Achtyes et al 2015;Gibbons et al 2012bGibbons et al , 2014.…”

“…The CAT-Mental Health (CAT-MH) study (Achtyes et al 2015;Gibbons et al 2012bGibbons et al , 2014 developed a bifactor-based CAT for ordinal response data (Gibbons et al 2007) and applied it to a 1,008-item bank consisting of 452 depression, 467 anxiety, and 89 bipolar items. We review only the results for depression.…”

Test-Retest Reliability of a Computerized Adaptive Depression Screener

“…Existing issues with depression screening and diagnosis in the ED may be overcome if one looks at the recent considerable progress made in the development of rapid screening (8) and measurement (9–10) of depression using computerized adaptive testing (CAT) based on multidimensional item response theory (IRT) (9,11,12). The advantages of this approach include: the use of large item banks (e.g.…”