From “below chance” to “a single error is one too many”: Evaluating various thresholds for invalid performance on two forced choice recognition tests

Erdődi, László A.

doi:10.1002/bsl.2609

Cited by 14 publications

(8 citation statements)

References 146 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In research settings, multivariate models of performance validity assessment enhance the research design by providing a more thorough sampling of the construct spread out across the testing session (Boone, 2009), as well as increasing the purity of criterion groups by determining the credibility of a given response set using a larger set of scores and thus, limiting a main source of error variance (i.e., random fluctuation in performance over time; Erdodi, 2023; Tyson et al, 2023). In an applied setting, a multivariate approach to PVTs improves the assessor’s confidence in the ultimate decision about the veracity of the profile, valid or invalid.…”

Section: Discussionmentioning

confidence: 99%

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

Tyson¹,

Shahein²

2023

Psychology & Neuroscience

View full text Add to dashboard Cite

Objective: This study was designed to explore the classification accuracy of various Word Choice Test (WCT) cutoffs. Method: Archival data were collected from a consecutive case series of a mixed sample of 79 patients clinically referred for neuropsychological assessment. Criterion groups were established using a combination of seven embedded validity indicators. Results: A WCT accuracy score of ≤45 emerged as the optimal accuracy cutoff (.47 sensitivity at .92 specificity). A time cutoff of ≥190" achieved a good combination of sensitivity (.40) and specificity (.93). Critical item analysis produced high specificity (.93-.98) but low and variable sensitivity (.19-.44). Combining the accuracy and time cutoffs resulted in perfect specificity, correctly classifying 91% of the sample. Conclusion: Imposing highly conservative accuracy cutoffs on the WCT unnecessarily sacrifices sensitivity. Recruiting alternative detection methods (completion time and critical items) seems a better approach, as it minimizes false positives while maximizing overall classification accuracy. Public Significance StatementThis study was designed to explore the classification accuracy of various cutoff scores on the WCT, a performance validity test used to differentiate credible and noncredible cognitive performance in patients undergoing neuropsychological assessment. Using a mixed clinical sample of 79 patients, our study identified optimal WCT cutoff scores for accuracy, time-to-complete, and critical item analysis. These findings demonstrate that multiple performance validity detection methods within the WCT can help maximize the diagnostic accuracy of neuropsychological assessment.

show abstract

Section: Discussionmentioning

confidence: 99%

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

Tyson¹,

Shahein²

2023

Psychology & Neuroscience

View full text Add to dashboard Cite

show abstract

“…Of the five participants that failed the TOMM T1 only (valid scores on the DS ACSS and WMT IR, DR, and CN indices), 0 participant scored in the below chance range (<18, per binomial theorem), and 0 participant scored in the chance level range (18-32, per binomial theorem). Of note, the absence of below chance scores should not be construed as evidence of credible performance, as below chance scores on PVTs have virtually zero negative predictive power (Erdodi, 2023).…”

Section: Pvt Invalid-two-indicator Criteriamentioning

confidence: 99%

Cross-validation and classification accuracy of informant report via the Dementia Severity Rating Scale (DSRS) for identifying examinee-generated performance and symptom invalidity.

Choudhury¹,

Christensen²,

Miller³

et al. 2023

Psychology & Neuroscience

View full text Add to dashboard Cite

Objective: Although informant report of examinee functioning has previously been associated with examinee-generated performance and cognitive symptom invalidity performance and symptoms validity test (PVT; SVT-C), extant data are limited, and empirically derived guidelines for interpreting informant-report validity are lacking. This study sought to (a) replicate the association of a widely used informant-report measure, the Dementia Severity Rating Scale (DSRS), and examinee-generated PVT/SVT-C in a larger/enriched sample and (b) recommend clinical cutoffs for informant-report invalidity. Method: Data were collected from 145 examinee–informant dyads who completed neuropsychological batteries as part of a routine workup in an epilepsy monitoring unit (EMU). PVT status was determined using two approaches based on combinations of three indicators, whereas SVT-C status was determined using three approaches based on combinations of two indicators. Receiver operating characteristic curves assessed the classification accuracy of the DSRS for discriminating PVT/SVT-C status. Results: The DSRS demonstrated acceptable accuracy in classifying PVT and SVT-C status across operationalizations (PVT AUCs = .71–.77; SVT-C AUCs = .71–.74). Replication sample data indicated the DSRS accurately classified PVT status area under the curves (AUCs = .73–.80), with a large-to-very large effect size across operationalizations (d = −.99 to −1.51). Replication sample data indicated the DSRS was inconsistent in accurately classifying SVT-C (AUCs = .63–.72), though a medium-to-large effect size was observed across operationalizations (d = −.71 to −.98). Conclusion: We replicate previously published associations of informant-reported examinee functioning and examinee-generated PVT/SVT-C. The results suggest empirically derived psychometric guidelines for interpreting informant reports as noncredible on an EMU and lay groundwork for future investigations of informant-report response validity.

show abstract

“…A. Erdodi, 2021;Etherton et al, 2006;Kim et al, 2010); CVLT-II FCR = forced-choice recognition trial of the California Verbal Learning Test-Second Edition (L. A L. A. Erdodi, 2021L. A. Erdodi, , 2023aMessa et al, 2022;Resch et al, 2022)…”

mentioning

confidence: 99%

Replicating the classification accuracy of the Verbal Paired Associates and Visual Reproduction recognition trials as embedded performance validity tests.

Crisan,

May,

Giromini

et al. 2024

Neuropsychology

Self Cite

View full text Add to dashboard Cite

Objective: This study was designed to replicate previous research on the clinical utility of the Verbal Paired Associates (VPA) and Visual Reproduction (VR) subtests of the WMS-IV as embedded performance validity tests (PVTs) and perform a critical item (CR) analysis within the VPA recognition trial. Method: Archival data were collected from a mixed clinical sample of 119 adults (M Age = 42.5, M Education = 13.9). Classification accuracy was computed against psychometrically defined criterion groups based on the outcome of various free-standing and embedded PVTs. Results: Age-corrected scaled scores ≤6 were specific (.89−.98) but had variable sensitivity (.36−.64). A VPA recognition cutoff of ≤34 produced a good combination of sensitivity (.46−.56) and specificity (.92−.93), as did a VR recognition cutoff of ≤4 (.48−.53 sensitivity at .86−.94 specificity). Critical item analysis expanded the VPA's sensitivity by 3.5%-7.0% and specificity by 5%-8%. Negative learning curves (declining output on subsequent encoding trials) were rare but highly specific (.99-1.00) to noncredible responding. Conclusions: Results largely support previous reports on the clinical utility of the VPA and VR as embedded PVTs. Sample-specific fluctuations in their classification accuracy warrant further research into the generalizability of the findings. Critical item analysis offers a cost-effective method for increasing confidence in the interpretation of the VPA recognition trial as a PVT.

show abstract

From “below chance” to “a single error is one too many”: Evaluating various thresholds for invalid performance on two forced choice recognition tests

Cited by 14 publications

References 146 publications

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

Combining accuracy scores with time cutoffs improves the specificity of the Word Choice Test.

Cross-validation and classification accuracy of informant report via the Dementia Severity Rating Scale (DSRS) for identifying examinee-generated performance and symptom invalidity.

Replicating the classification accuracy of the Verbal Paired Associates and Visual Reproduction recognition trials as embedded performance validity tests.

Contact Info

Product

Resources

About