Estimating the accuracy of neurocognitive effort measures in the absence of a “gold standard”.

Abstract

Psychologists frequently use symptom validity tests (SVTs) to help determine whether evaluees' test performance or reported symptoms accurately represent their true functioning and capability. Most studies evaluating the accuracy of SVTs have used either known-group comparisons or simulation designs, but these approaches have well-known limitations (potential misclassifications or lack of ecological validity). This study uses latent class modeling (LCM) implemented in a Bayesian framework to estimate SVT classification accuracy based on data obtained from real-life forensic evaluations. We obtained archival data from 1,301 outpatient evaluees who underwent testing with the Computerized Assessment of Response Bias (CARB), the Test of Memory Malingering (TOMM), and the Word Memory Test (WMT) in a forensic evaluation context. Under various data models, Markov chain Monte Carlo methods implemented via WinBUGS converged to target distributions that permitted Bayesian estimates of SVT accuracy. Under the most plausible model (conditional dependence in test results), classification accuracies (expressed as area under the "trapezoidal" receiver operating characteristic curve ± standard deviation) were as follows: CARB = 0.765 ± 0.030, WMT = 0.929 ± 0.020, and TOMM = 0.771 ± 0.034. At decision thresholds that hold false positive rates at 0.02, the SVTs would detect invalid responses (true positives) at rates of approximately 35%, 65%, and 49%, respectively, for the 3 tests. Though LCM methods have limitations, this study suggests that they offer an approach to SVT evaluation that avoids methodological pitfalls of known-group research designs while retaining ecological validity that is absent in simulation studies.

Diagnostic Accuracy of a Bayesian Latent Group Analysis for the Detection of Malingering-Related Poor Effort

“…To this respect, we also acknowledge that the WMT classification results might have been indeed different (e.g., lower false positive rate) if applying all WMT subtests and analyzing the GMIP profile. Nevertheless, Mossman et al (2012) recently obtained high overall diagnostic accuracy (i.e., AUC = .929 ± .020) using the same WMT total raw scores as we did under a comparable Bayesian framework. Therefore, the reason for the high false positive rate observed in our study needs to be further investigated.…”

“…In practice, the occurrence of both false positive and false negative errors may lead to incorrect legal decisions or delay public support for people who are truly in need (Franzen, Iverson, & McCracken, 1990;Mossman, Wygant, & Gervais, 2012). To prevent potentially negative consequences of misclassification, malingering researchers have proposed and refined different malingering diagnostic criteria (Boone, 2007;Slick, Sherman, & Iverson, 1999).…”

“…In recent years, statistical methods used to detect poor effort in neuropsychological practice and research included odds and likelihood ratios (Bieliauskas, Fastenau, Lacy, & Roper, 1997;Weinborn, Orr, Woods, Conover, & Feix, 2003), the aggregation across multiple indicators Larrabee, Millis, & Meyers, 2008), Bayesian average modeling Millis & Volinsky, 2001;Wolfe et al, 2010), and Bayesian latent class modeling (Mossman et al, 2012). In this context, we recently tested a Bayesian latent group analysis as a method that may help to improve classification accuracy of effort testing in various neuropsychological settings (Ortega, Wagenmakers, Lee, Markowitsch, & Piefke, 2012).…”

“…This decision relates to several well-designed studies that also used only the WMT primary effort subtests as a screening effort measure (e.g., Batt, Shores, & Chekaluk, 2008;Greiffenstein, Greve, Bianchini, & Baker, 2008;Mossman et al, 2012). The rationale behind this methodological decision was to ensure that all effort measures share a comparable theoretical construct targeting memory recognition.…”

“…For this reason, all accuracy estimates were obtained using a single indicator from each effort measure in all analyses. Following the procedure used by Mossman et al (2012) the WMT recognition subtests were transformed into a single effort indicator (i.e., WMT total score). The WMT IR, DR, and CNS percentages were first converted into raw scores, and then averaged to obtain a WMT total score.…”