Early bird fails the PVT? The effects of timing artifacts on performance validity tests.

Abstract

This study was designed to investigate the effects of timing on the likelihood of failing the Recognition Memory Test-Words (RMT) and Word Choice Test (WCT). The RMT and WCT were administered in counterbalanced order either at the beginning (Time 1) or at the end (Time 2) of a test battery to a mixed clinical sample of 196 patients (M = 44.5 years, 55.1% female) medically referred for neuropsychological evaluation. The risk of failing the accuracy score was higher at Time 1 on both the RMT (relative risk [RR]: 1.44-1.64) and the WCT (RR: 1.21-1.50) across a range of cutoffs. Likewise, the risk of failing the time-to-completion score was higher at Time 1 on both the RMT (RR: 1.30-1.94) and the WCT (RR: 1.58-3.75). Established cutoffs failed to reach specificity standards at Time 1; more liberal cutoffs cleared specificity thresholds at Time 2. According to our findings, the RMT and WCT may be prone to false-positive errors at Time 1. Conversely, when administered at Time 2, existing cutoffs may have lower sensitivity, but they are highly specific to invalid performance. Timing should be considered during both test selection and the interpretation of RMT and WCT scores. Using conservative cutoffs for morning administrations and liberal cutoffs for afternoon administrations may be necessary to neutralize timing artifacts. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Flipping the Script: Measuring Both Performance Validity and Cognitive Ability with the Forced Choice Recognition Trial of the RCFT

“… Note . EVI: Embedded validity indicators; PVT: Performance Validity Test; RCFT: Rey Complex Figure Test; Y/N Rec: Yes/No recognition raw score; FCR: Forced Choice Recognition raw score; TOMM-1: Trial 1 on the Test of Memory Malingering ( Denning, 2012 ; Fazio et al., 2017 ; Greve et al., 2006 , 2009 ; Jones, 2013 ; Kulas et al., 2014 ; Martin et al., 2019; Powell et al., 2004 ; Rai & Erdodi, 2019 ; Webber et al., 2018 ); WCT: Word Choice Test [Fail defined as accuracy score ≤47 ( Barhon et al., 2015 ; Davis, 2014 ; Erdodi, Kirsch, et al., 2014 ; Pearson, 2009 ) or time-to-completion ≥156 seconds (Erdodi & Lichtenstein, 2020; Erdodi, Tyson, et al., 2017 ; Zuccato et al., 2018 )]; EI-5 MEM: Erdodi Index Five – Memory ( Fail defined as ≥4); EI-5 PSP: Erdodi Index Five – Processing Speed ( Fail defined as ≥4); BR Fail: Base rate of failure (% of the sample that failed a given cutoff); SENS: Sensitivity; SPEC: Specificity. …”

“…Lowering the cutoff to 17 resulted in trivial gains in specificity (.84-.97) but a precipitous drop in sensitivity (.29-.54). Making the cutoff even more conservative ( 16) (Denning, 2012;Fazio et al, 2017;Greve et al, 2006Greve et al, , 2009Jones, 2013;Kulas et al, 2014;Martin et al, 2020;Powell et al, 2004;; WCT: Word Choice Test [Fail defined as accuracy score 47 (Barhon et al, 2015;Davis, 2014;Pearson, 2009) or time-to-completion !156 seconds (Erdodi & Lichtenstein, 2021;Zuccato et al, 2018)]; EI-5 MEM: Erdodi Index Five -Memory (Fail defined as !4); EI-5 PSP: Erdodi Index Five -Processing Speed (Fail defined as !4). (Denning, 2012;Fazio et al, 2017;Greve et al, 2006Greve et al, , 2009Jones, 2013;Kulas et al, 2014;Martin et al, 2019;Powell et al, 2004;; WCT: Word Choice Test [Fail defined as accuracy score 47 (Barhon et al, 2015;Davis, 2014;Pearson, 2009) or time-to-completion !156 seconds (Erdodi & Lichtenstein, 2020;Zuccato et al, 2018)]; EI-5 MEM: Erdodi Index Five -Memory (Fail defined as !4); EI-5 PSP: Erdodi Index Five -Processing Speed (Fail defined as !4); BR Fail: Base rate of failure (% of the sample that failed a given cutoff); SENS: Sensitivity; SPEC: Specificity.…”

Introducing a Five-Variable Psychiatric Screener based on the Visual Analog Scale (V-5).

“…Diagnoses were determined by the assessing clinicians based on Diagnostic and Statistical Manual of Mental Disorders , Fifth Edition ( DSM-5 ; APA, 2013) criteria using available medical records, clinical interviews, and psychometric data. A portion of the sample was used in previous publications focused on different topics (Erdodi, 2019; Erdodi et al, 2017; Erdodi, Dunn, et al, 2018; Erdodi, Hurtubise, et al, 2018; Erdodi & Lichtenstein, 2017; Erdodi, Pelletier, & Roth, 2018; Erdodi & Roth, 2017; Erdodi, Sagar, et al, 2018; Erdodi, Seke, et al, 2017; Erdodi, Tyson, et al, 2017, 2018; Rai & Erdodi, 2019; Zuccato et al, 2018). The only inclusion criterion was having data available on the Five-Variable Psychiatric Screener (V-5).…”

“…Quantifying key constructs such as depression, anxiety, or somatic concerns is necessary for effective large-scale screening efforts to establish a diagnosis and track progress during psychotherapy or pharmacological interventions. Measuring treatment outcome requires multiple assessments, raising both methodological (time-related changes in the target construct; Erdodi & Lajiness-O’Neill, 2014; Erdodi, Lajiness-O’Neill, & Saules, 2010; Zuccato, Tyson, & Erdodi, 2018) and logistical (high cost of repeated testing) issues. However, frequent (i.e., several times a day) evaluation of depression may be necessary to accurately characterize a patient’s mood state during periods of high volatility, such as acute inpatient psychiatric treatment (Hall, Sing, & Romanoski, 1991).…”

Handedness and performance validity test performance.

“…Inclusion criteria included participants who were less than 70 years of age and had data available for the RMT and WCT. This sample has been used in previous publications (Erdodi, Sagar et al, 2018; Erdodi, Seke et al, 2017; Erdodi, Tyson et al, 2018; Erdodi, Tyson et al, 2017; Zuccato, Tyson, & Erdodi, 2018).…”