Development of embedded performance validity indicators in the NIH Toolbox Cognitive Battery.

Abeare, Christopher A.; Erdődi, László A.; Messa, Isabelle; Terry, Douglas P.; Panenka, William J.; Silverberg, Noah D.

doi:10.1037/pas0000958

Cited by 14 publications

(6 citation statements)

References 44 publications

(56 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, there was no difference between these two groups for BR Fail on the WCT. These findings are consistent with previous reports ( Abeare, Erdodi, et al., 2020 ; Erdodi, Hurtubise, et al., 2018 ), and they challenge the prominence of the examinees’ external incentive status in diagnostic models for malingering (APA, 2013; Erdodi et al., 2018 ; Slick et al., 1999 ). While the new RCFT FCR trial was more sensitive to the effect of incentive status than the RCFT Yes/No Recognition trial ( Figure 1 ), it must be noted that incentive to appear impaired and elevated risk of genuine neuropsychological deficits were conflated in our clinical sample.…”

Section: Incidental Findingssupporting

confidence: 92%

“…The VPA-3 is analogous to the WAIS-IV index scores (Verbal Comprehension, Perceptual Reasoning, Working Memory and Processing Speed) as it combines information from multiple tests into a single-number summary of the broader construct. Multivariate measurement models based on aggregating different sources of data have been shown to be superior to single test scores ( Abeare, Erdodi, et al., 2020 ; Pearson, 2009 ; Tyson et al., 2018 ). For the specific purpose of this study, the VPA-3 served as the criterion for calibrating the RCFT FCR as a measure of visual recognition memory (i.e., ability test).…”

Section: Methodsmentioning

confidence: 99%

“…Moreover, emerging evidence suggests that the instructions given to examinees have a weak effect on the credibility of their response sets overall ( Abeare, Hurtubise, et al., 2020 ; Niesten et al., 2017 ). In other words, reminding real-world patients to provide valid data does not assure that outcome.…”

Section: Reflections On Control Group Methodologiesmentioning

confidence: 99%

“…Likewise, past studies showed that a variable proportion of student volunteers who were assigned to the control condition and asked to demonstrate their best ability in academic research settings failed PVTs ( An et al., 2012 ; DeRight & Jorgensen, 2015 ; Ross et al., 2016 ; Roye et al., 2019 ; Santos et al., 2014 ; Silk-Eglit et al., 2014 ). Ironically, participants assigned to the experimental malingering condition also occasionally demonstrated intact cognitive ability – in other words, they failed at failing ( Abeare et al., 2020 ).…”

Section: Reflections On Control Group Methodologiesmentioning

confidence: 99%

See 3 more Smart Citations

Flipping the Script: Measuring Both Performance Validity and Cognitive Ability with the Forced Choice Recognition Trial of the RCFT

et al. 2021

View full text Add to dashboard Cite

In this study we attempted to replicate the classification accuracy of the newly introduced Forced Choice Recognition trial (FCR) of the Rey Complex Figure Test (RCFT) in a clinical sample. We administered the RCFT FCR and the earlier Yes/No Recognition trial from the RCFT to 52 clinically referred patients as part of a comprehensive neuropsychological test battery and incentivized a separate control group of 83 university students to perform well on these measures. We then computed the classification accuracies of both measures against criterion performance validity tests (PVTs) and compared results between the two samples. At previously published validity cutoffs (≤16 & ≤17), the RCFT FCR remained specific (.84–1.00) to psychometrically defined non-credible responding. Simultaneously, the RCFT FCR was more sensitive to examinees’ natural variability in visual-perceptual and verbal memory skills than the Yes/No Recognition trial. Even after being reduced to a seven-point scale (18-24) by the validity cutoffs, both RCFT recognition scores continued to provide clinically useful information on visual memory. This is the first study to validate the RCFT FCR as a PVT in a clinical sample. Our data also support its use for measuring cognitive ability. Replication studies with more diverse samples and different criterion measures are still needed before large-scale clinical application of this scale.

show abstract

Section: Incidental Findingssupporting

confidence: 92%

Section: Methodsmentioning

confidence: 99%

Section: Reflections On Control Group Methodologiesmentioning

confidence: 99%

Section: Reflections On Control Group Methodologiesmentioning

confidence: 99%

See 2 more Smart Citations

Flipping the Script: Measuring Both Performance Validity and Cognitive Ability with the Forced Choice Recognition Trial of the RCFT

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Our metrics offer a method to examine gradients of performance validity, provide multiple measures across individual tests, and integrate response speed and item-level psychometrics. Similarly, item-level metrics from another computerized battery, the NIH Toolbox-Cognition, have recently shown promise predicting externally validated PVTs in a mild traumatic brain injury sample (Abeare et al, 2021). Such metrics may facilitate administration of neurocognitive tests through telehealth or other remote applications, as well as the integration of cognitive testing into "big data" efforts, including genomics.…”

Section: Discussionmentioning

confidence: 99%

Development and application of novel performance validity metrics for computerized neurocognitive batteries

Scott

Moore

Roalf

et al. 2022

J Int Neuropsychol Soc

View full text Add to dashboard Cite

Objectives: Data from neurocognitive assessments may not be accurate in the context of factors impacting validity, such as disengagement, unmotivated responding, or intentional underperformance. Performance validity tests (PVTs) were developed to address these phenomena and assess underperformance on neurocognitive tests. However, PVTs can be burdensome, rely on cutoff scores that reduce information, do not examine potential variations in task engagement across a battery, and are typically not well-suited to acquisition of large cognitive datasets. Here we describe the development of novel performance validity measures that could address some of these limitations by leveraging psychometric concepts using data embedded within the Penn Computerized Neurocognitive Battery (PennCNB). Methods: We first developed these validity measures using simulations of invalid response patterns with parameters drawn from real data. Next, we examined their application in two large, independent samples: 1) children and adolescents from the Philadelphia Neurodevelopmental Cohort (n = 9498); and 2) adult servicemembers from the Marine Resiliency Study-II (n = 1444). Results: Our performance validity metrics detected patterns of invalid responding in simulated data, even at subtle levels. Furthermore, a combination of these metrics significantly predicted previously established validity rules for these tests in both developmental and adult datasets. Moreover, most clinical diagnostic groups did not show reduced validity estimates. Conclusions: These results provide proof-of-concept evidence for multivariate, data-driven performance validity metrics. These metrics offer a novel method for determining the performance validity for individual neurocognitive tests that is scalable, applicable across different tests, less burdensome, and dimensional. However, more research is needed into their application.

show abstract

Assessing Negative Response Bias with the Millon Clinical Multiaxial Inventory-IV (MCMI-IV): a Review of the Literature

Choca

Pignolo

2022

Psychol. Inj. and Law

View full text Add to dashboard Cite

Development of embedded performance validity indicators in the NIH Toolbox Cognitive Battery.

Cited by 14 publications

References 44 publications

Flipping the Script: Measuring Both Performance Validity and Cognitive Ability with the Forced Choice Recognition Trial of the RCFT

Flipping the Script: Measuring Both Performance Validity and Cognitive Ability with the Forced Choice Recognition Trial of the RCFT

Development and application of novel performance validity metrics for computerized neurocognitive batteries

Assessing Negative Response Bias with the Millon Clinical Multiaxial Inventory-IV (MCMI-IV): a Review of the Literature

Contact Info

Product

Resources

About