Embedded Performance Validity Measures with Postdeployment Veterans: Cross-Validation and Efficiency with Multiple Measures

Abstract

Embedded validity measures support comprehensive assessment of performance validity. The purpose of this study was to evaluate the accuracy of individual embedded measures and to reduce them to the most efficient combination. The sample included 212 postdeployment veterans (average age = 35 years, average education = 14 years). Thirty embedded measures were initially identified as predictors of Green's Word Memory Test (WMT) and were derived from the California Verbal Learning Test-Second Edition (CVLT-II), Conners' Continuous Performance Test-Second Edition (CPT-II), Trail Making Test, Stroop, Wisconsin Card Sorting Test-64, the Wechsler Adult Intelligence Scale-Third Edition Letter-Number Sequencing, Rey Complex Figure Test (RCFT), Brief Visuospatial Memory Test-Revised, and the Finger Tapping Test. Eight nonoverlapping measures with the highest area-under-the-curve (AUC) values were retained for entry into a logistic regression analysis. Embedded measure accuracy was also compared to cutoffs found in the existing literature. Twenty-one percent of the sample failed the WMT. Previously developed cutoffs for individual measures showed poor sensitivity (SN) in the current sample except for the CPT-II (Total Errors, SN = .41). The CVLT-II (Trials 1-5 Total) showed the best overall accuracy (AUC = .80). After redundant measures were statistically eliminated, the model included the RCFT (Recognition True Positives), CPT-II (Total Errors), and CVLT-II (Trials 1-5 Total) and increased overall accuracy compared with the CVLT-II alone (AUC = .87). The combination of just 3 measures from the CPT-II, CVLT-II, and RCFT was the most accurate/efficient in predicting WMT performance.

Analysis of performance on a modified Wisconsin Card Sorting Test for the military

“…Participants then sat at a standard desk and completed the computerized map task as if they were informing, yet Wisconsin Card Sorting Test Results Table I shows the summary statistics of the map task results. Performance on most measures is consistent with results from the original WCST including a sample of veterans of similar age to our participants (Shan et al, 2008;Shura et al, 2015). However, percentage of non-perseverative error was higher than that of previous studies (Shan et al, 2008): mean of 20.86 per cent (SD = 13.57 per cent).…”

CVLT-II Forced Choice Recognition Trial as an Embedded Validity Indicator: A Systematic Review of the Evidence

“…However, of the 328 patients across six studies who failed the Test of Memory Malingering (TOMM) at standard cutoffs, 65.6% failed FCR (Axelrod & Schutte, 2011; Egeland, Andersson, Sundseth, & Schanke, 2015; Heyanka et al, 2015; Jak et al, 2015; Orff et al, 2015; Sugarman & Axelrod, 2015). In contrast, only 30.9% of the 249 patients across four studies who failed the Word Memory Test (WMT) at standard cutoffs scored ≤14 on FCR (Donders & Strong, 2011; Erdodi, Kirsch, et al, 2014; Erdodi, Roth, et al, 2014; Heyanka et al, 2015; Shura, Miskey, Rowland, Yoash-Gantz, & Denning, 2015).…”

The Post‐Deployment Mental Health (PDMH) study and repository: A multi‐site study of US Afghanistan and Iraq era veterans

“…The BDS-II showed improved psychometric properties and was significantly more reliable than the BDS. Shura, Rowland, & Yoash-Gantz, 2015 Sample evaluated 164 subjects. Purpose was to evaluate the factor structure and construct validity of the BDS-II. To recruit a target population Shura, Miskey, Rowland, Yoash-Gantz, & Denning, 2016 Compared 44 “failed-validity” and 168 “passed-validity” groups. Purpose was to evaluate the accuracy of individual embedded validity measures and to reduce them to the most efficient combination to improve comprehensive assessment of performance validity. To recruit a target population Neurosteroids and other therapeutics Marx et al, 2009 Proof-of-concept randomized controlled trial of adjunctive pregnenolone for cognitive and negative symptoms in patients with schizophrenia or schizoaffective disorder.…”