Symptom and performance validity in samples of adults at clinical evaluation of ADHD: a replication study using machine learning algorithms

Abstract

Introduction:Research has shown non-trivial base rates of noncredible symptom report and performance in the clinical evaluation of attention-deficit/hyperactivity disorder (ADHD) in adulthood. The goal of this study is to estimate and replicate base rates of symptom and performance validity test failure in the clinical evaluation of adult ADHD and derive prediction models based on routine clinical measures. Methods: This study reuses data of a previous publication of 196 adults seeking ADHD assessment and replicates the findings on an independent sample of 700 adults recruited in the same referral context. Measures of symptom and performance validity (one SVT, two PVTs) were applied to estimate base rates. Prediction models were developed using machine learning. Results: Both samples showed substantial rates of noncredible symptom report (one SVT failure: 35.7% -36.6%), noncredible test performance (one PVT failure: 32.1% -49.3%; two PVT failures: 18.9% -27.3%), or both (each one SVT and PVT failure: 13.3% -22.4%; one SVT and two PVT failures: 9.7% -13.7%). Machine learning algorithms resulted in generally moderate to weak prediction models, with advantages of the reused sample compared to the independent replication sample. Associations between measures of symptom and performance validity were negligible to small. Conclusions: This study highlights the necessity to include measures of symptom and performance validity in the clinical evaluation of adult ADHD. Further, this study demonstrates the difficulty to characterize the group failing symptom or performance validity assessment.

Cognitive Underperformance in a Mixed Neuropsychiatric Sample at Diagnostic Evaluation of Adult ADHD

“…However, despite reaching significance, the effect may not be of clinical relevance, as the upper bound of the CI falls close to zero, which may indicate that a trivial effect cannot be excluded with sufficient certainty. PVT failure does not seem to be explained sufficiently well by clinical characteristics, which is in line with earlier findings on large samples of a comparable referral context (see for example, [3,28]). In the present study, individuals referred to the outpatient diagnostic unit had to accept several months of waiting time before they were invited for an assessment, which suggests that "help-seeking behavior" may be a reason for noncredible data in at least some of the cases in order to convince the clinician of their experienced problems and need for support.…”

“…Based on the same criterion of determining invalid cognitive performance (≥2 PVT failures), Mascarenhas et al [27] found a base rate of 9.4% of invalid performance in a large sample (n = 1045) of post-secondary students who underwent a comprehensive psychoeducational evaluation. Hirsch et al [3,28] reported failure rates on one PVT (Amsterdam Short Term Memory Test, ASTM) between 32.1 and 49.3% on samples of 196 and 700 adults with ADHD, respectively. When applying a stricter criterion for noncredible performance of positive results on two PVTs on the same samples, base rates dropped to 18.9-27.3% [3,28].…”

“…Hirsch et al [3,28] reported failure rates on one PVT (Amsterdam Short Term Memory Test, ASTM) between 32.1 and 49.3% on samples of 196 and 700 adults with ADHD, respectively. When applying a stricter criterion for noncredible performance of positive results on two PVTs on the same samples, base rates dropped to 18.9-27.3% [3,28]. Hence, differences in PVT methodology, rigorousness of determining invalid cognitive performance, and sample composition across studies are, obviously, major factors for the heterogeneity of base rate estimations.…”

“…performance can be manifold between and within individuals, including conscious and unconscious forms, and often depend on the heterogeneous composition of the sample, their motivation, and the context of the assessment [1][2][3][4]. While the underlying reason for noncredible symptom reports and performance may be difficult to determine on the individual level, a non-trivial proportion may be motivated to deliberately feign ADHD in order to obtain access to external or internal benefits.…”