Validity of informant report interpretations: Role of examinee performance and symptom invalidity.

Abstract

Performance validity tests (PVTs) and symptom validity tests (SVTs) detect inaccuracies in examinee-completed measures, though methods for assessing the accuracy of informant reports—which may be inaccurate due to examinee deception, motivation for external incentives (e.g., disability payments), or attempts to validate examinee experiences—remain underexplored. We used a sample of 72 veteran-informant dyads undergoing evaluation of possible epilepsy-related neurocognitive disorder to assess the association between examinee response invalidity (i.e., performance and/or symptom invalidity) and informant report measures. Examinees completed PVTs, SVTs, cognitive, and self-report measures. Informants completed measures on examinee functioning and their own caregiver burden. Performance invalidity was defined as failure on two or more PVTs. Symptom invalidity for psychopathology symptom reports (SVT-P) and cognitive/somatic symptom reports (SVT-CS) were separately defined via above-threshold scores on two or more SVT criterion. Independent samples t tests demonstrated the associations of the PVT, SVT-CS, and SVT-P groups with informant-report measures. Informants for examinees with performance invalidity reported worse functioning in the examinee than informants for examinees in the valid performance group (medium-large effect sizes). Symptom validity status (for both SVT-CS and SVT-P) was meaningfully but less strongly related to informant-reported examinee functioning (small–medium effect sizes). Neither performance nor symptom invalidity was meaningfully related to informant-reported caregiver burden (negligible effect sizes). Informant reports for examinees with response invalidity should be interpreted with caution.

Cross-validation and classification accuracy of informant report via the Dementia Severity Rating Scale (DSRS) for identifying examinee-generated performance and symptom invalidity.

“…As an extension of this work and consistent with our hypotheses, in this study, the DSRS exhibited acceptable accuracy in classifying examineegenerated performance invalidity using the twocriteria/composite approach to operationalize PVT invalidity, with a cut-score of DSRS ≥ 22 yielding .93 specificity. In contrast, Webber and colleagues did not previously identify statistically significant associations between informant report of examinee functioning and SVT-C status, though small-to-medium effect sizes were observed (Webber et al, 2022). In the present study, the DSRS exhibited acceptable accuracy in classifying examinee-generated symptom invalidity using the two-criteria/composite approach to operationalize SVT-C invalidity, with a cutscore of DSRS ≥ 22 yielding .90 specificity.…”

“…Although the results of this study highlighted the association between examinee-generated performance/symptom invalidity and informant-report data, the study by Webber et al (2022) is limited in several ways. First, the total sample size was relatively limited ( n = 72).…”

“…Such attenuations in statistical power and stability of estimates necessitate replication in a larger sample that may be better equipped to reliably estimate the effect size magnitude and detect a statistically significant signal. The relatively smaller effect sizes of the cognitive/somatic SVT group variable (Cohen’s d = .39–.52) compared to PVT group status variable (Cohen’s d = .56–.98) in Webber et al (2022) may also reflect that use of both cognitive and somatic symptom dimensions in the variable operationalization, necessitating replication of the aforementioned associations with a cognitive SVT group variable that is minimally adulterated by somatic symptom invalidity.…”

“…In an effort to address this critical gap, Webber et al (2022) conducted the first and only known study on the association of examinee-completed PVTs/SVTs and informant report of examinee functioning in a sample of 72 individuals undergoing neuropsychological assessment in an epilepsy monitoring unit (EMU) sample. The results demonstrated that informants reported worse functioning for examinees with performance and symptom invalidity derived from standardized tests than for examinees with valid test performances.…”

Assessment of response bias in personality disorder research.

“…D. Miller & Lynam, 2015;Quilty et al, 2018;Vazire, 2006Vazire, , 2010. However, informants may overreport alongside examinees in the context of external incentives (Webber et al, 2022) and even in anonymous research contexts (Quilty et al, 2018). Unfortunately, very little attention has been given to developing methods to screen for invalid informant data.…”