Psychometric implications of failure on one performance validity test: a cross-validation study to inform criterion group definition

Rhoads, Tasha; Neale, Alec C; Resch, Zachary J; Cohen, Cari D.; Keezer, Richard D; Cerny, Brian M.; Jennette, Kyle J; Ovsiew, Gabriel P.; Soble, Jason R.

doi:10.1080/13803395.2021.1945540

Cited by 44 publications

(26 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Patients with one or fewer PVT failures constituted the valid group ( n = 274), whereas those with two or more PVT failures constituted the invalid group ( n = 43). This grouping method is consistent with current practice standards as well as the revised criteria for identifying invalid neuropsychological test profiles (Sherman et al, 2020; Sweet et al, 2021), and is further empirically supported by recent findings suggesting that the inclusion of individuals with a single PVT failure as performing validly is an acceptable PVT research methodology (Jennette et al, 2021; Rhoads et al, 2021). Of the 274 in the valid group, 226 cases met formal Diagnostic and Statistical Manual of Mental Disorders-5th Edition (DSM-5; American Psychiatric Association, 2013) criteria for ADHD and were therefore assigned to the valid-confirmed ADHD group.…”

Section: Methodssupporting

confidence: 77%

A comparison of embedded validity indicators from the Stroop Color and Word Test among adults referred for clinical evaluation of suspected or confirmed attention-deficit/hyperactivity disorder.

Khan¹,

Rauch²,

Obolsky³

et al. 2022

Psychological Assessment

Self Cite

View full text Add to dashboard Cite

This study investigated the utility of four Stroop Color and Word Test (SCWT) indices, including the raw score and T score for the word reading (WR) and color naming (CN) trials, as embedded performance validity tests (PVTs) within a sample referred for evaluation of suspected or known attention-deficit/hyperactivity disorder (ADHD). Data were analyzed from a final sample of 317 patients consecutively referred for ADHD evaluation, which was divided into groups with invalid (n = 43; 14%) and valid neuropsychological test performance (n = 274; 86%). A subset of the valid group with confirmed ADHD diagnoses (n = 226; 71%) were also analyzed separately. Classification accuracy for the overall valid sample was in the acceptable range (AUCs = .757–.794), with optimal cut scores of WR raw ≤75 (54% sensitivity/90% specificity), WR T score ≤28 (54% sensitivity/88% specificity), CN raw ≤57 (42% sensitivity/90% specificity), and CN T score ≤30 (40% sensitivity/90% specificity). Classification accuracy was also in the acceptable range for the ADHD-confirmed subgroup (AUCs = .750–.790), with optimal cut scores of WR Raw ≤75 (54% sensitivity/89% specificity), WR T score ≤28 (54% sensitivity/87% specificity), CN Raw ≤57 (42% sensitivity/90% specificity), and CN T score ≤30 (40% sensitivity/90% specificity). These findings indicate that embedded PVTs derived from the SCWT, particularly those derived from the WR trial, are effective measures for determining validity status in samples with suspected or confirmed ADHD.

show abstract

Section: Methodssupporting

confidence: 77%

A comparison of embedded validity indicators from the Stroop Color and Word Test among adults referred for clinical evaluation of suspected or confirmed attention-deficit/hyperactivity disorder.

Khan¹,

Rauch²,

Obolsky³

et al. 2022

Psychological Assessment

Self Cite

View full text Add to dashboard Cite

show abstract

“…All patients were administered a battery of seven freestanding and embedded PVTs that previously have been cross-validated in ADHD samples or ADHD-predominant mixed clinical samples, including the Dot Counting Test (Abramson et al, 2021; Boone et al, 2002); Rey 15-Item Test Recall/Recognition (Ashendorf et al, 2021), Rey Auditory Verbal Learning Test Effort Score (Pliskin et al, 2021), Brief Visuospatial Memory Test-Revised Recognition Discrimination (Bailey et al, 2018; Resch et al, 2020), Stroop Test Word Reading T-Score (White, Korinek, et al, 2020); Trail Making Test - Part A (White, Korinek, et al, 2020), and Verbal Fluency (F/A/S; White, Korinek, et al, 2020). Consistent with current empirical findings and neuropsychology practice standards, those with ≤1 criterion PVT failures were classified into the valid group and those with ≥2 failures were assigned to the invalid group (e.g., Critchfield et al, 2019; Jennette et al, 2021; Larrabee, 2014; Rhoads et al, 2021; Schroeder, Martin, Heinrichs, & Baade, 2019; Soble et al, 2020).…”

Section: Methodsmentioning

confidence: 99%

Cross-Validation of Multiple WAIS-IV Digit Span Embedded Performance Validity Indices Among a Large Sample of Adult Attention Deficit/Hyperactivity Disorder Clinical Referrals

Bing-Canar

Phillips

Shields

et al. 2022

Journal of Psychoeducational Assessment

Self Cite

View full text Add to dashboard Cite

This study investigated the utility of four WAIS-IV Digit Span (DS) indices (traditional Reliable Digit Span [RDS], RDS-Working Memory [RDS-WM], RDS-Revised [RDS-R], and DS Age-Corrected Scaled Score [ACSS]) as embedded performance validity tests (PVTs) among a sample of 342 consecutive adults referred for neuropsychological evaluation of ADHD. All DS indices had acceptable classification accuracy (areas under the curve: .73–.76) for detecting invalid performance with optimal cut-scores of RDS ≤7 (35% sensitivity/93% specificity), RDS-WM ≤7 (56% sensitivity/86% specificity), RDS-R ≤12 (48% sensitivity/85% specificity), and ACSS ≤7 (46% sensitivity/87% specificity). Although all indices were able to detect invalid performance, DS indices incorporating the more complex working memory trials of the task yielded the best accuracy for identification of invalid test performance among adults referred for ADHD evaluation.

show abstract

“…The criterion PVTs were identical across samples, with one exception in the VA sample—the Word Memory Test (WMT; Green, 2003)—and one in the AMC sample—the Medical Symptom Validity Test (MSVT; Green, 2004). We established the test-taking validity status of each patient, based on current neuropsychological practice guidelines and empirical findings, such that patients with ≤1 criterion PVT failure were classified as having a valid neuropsychological test performance and those with ≥2 PVT failures were classified as demonstrating an invalid test performance (Critchfield et al, 2019; Rhoads, Neale, 2021; Soble, Alverson et al, 2020; Sweet et al, 2021).…”

Section: Methodsmentioning

confidence: 99%

Comparing the Independent and Aggregated Accuracy of Trial 1 and the First 10 TOMM Items for Detecting Invalid Neuropsychological Test Performance Across Civilian and Veteran Clinical Samples

et al. 2022

Self Cite

View full text Add to dashboard Cite

Previous studies support using two abbreviated tests of the Test of Memory Malingering (TOMM), including (a) Trial 1 (T1) and (b) the number of errors on the first 10 items of T1 (T1e10), as performance validity tests (PVTs). In this study, we examined the independent and aggregated predictive utility of TOMM T1 and T1e10 for identifying invalid neuropsychological test performance across two clinical samples. We employed cross-sectional research to examine two independent and demographically diverse mixed samples of military veterans and civilians (VA = 108; academic medical center = 234) of patients who underwent neuropsychological evaluations. We determined validity groups by patient performance on four independent criterion PVTs. We established concordances between passing/failing the TOMM T1e10 and T1, followed by logistic regression to determine individual and aggregated accuracy of T1e10 and T1 for predicting validity group membership. Concordance between passing T1e10 and T1 was high, as was overall validity (87–98%) across samples. By contrast, T1e10 failure was more highly concordant with T1 failure (69–77%) than with overall invalidity status (59–60%) per criterion PVTs, whereas T1 failure was more highly concordant with invalidity status (72–88%) per criterion PVTs. Logistic regression analyses demonstrated similar results, with T1 accounting for more variance than T1e10. However, combining T1e10 and T1 accounted for the most variance of any model, with T1e10 and T1 each emerging as significant predictors. TOMM T1 and, to a lesser extent, T1e10 were significant predictors of independent criterion-derived validity status across two distinct clinical samples, but they did not offer improved classification accuracy when aggregated.

show abstract

Psychometric implications of failure on one performance validity test: a cross-validation study to inform criterion group definition

Cited by 44 publications

References 43 publications

A comparison of embedded validity indicators from the Stroop Color and Word Test among adults referred for clinical evaluation of suspected or confirmed attention-deficit/hyperactivity disorder.

A comparison of embedded validity indicators from the Stroop Color and Word Test among adults referred for clinical evaluation of suspected or confirmed attention-deficit/hyperactivity disorder.

Cross-Validation of Multiple WAIS-IV Digit Span Embedded Performance Validity Indices Among a Large Sample of Adult Attention Deficit/Hyperactivity Disorder Clinical Referrals

Comparing the Independent and Aggregated Accuracy of Trial 1 and the First 10 TOMM Items for Detecting Invalid Neuropsychological Test Performance Across Civilian and Veteran Clinical Samples

Contact Info

Product

Resources

About