A multi-study examination of performance validity in undergraduate research participants

Flipping the Script: Measuring Both Performance Validity and Cognitive Ability with the Forced Choice Recognition Trial of the RCFT

“…This surprising finding underlines the importance of empirically verifying even intuitive and logically appealing assumptions. Our results supported, instead, previously voiced theoretical concerns ( Giromini et al., 2019 ; McWhirter et al., 2019 ; van Helvoort et al., 2019 ) and published data ( Abeare et al., 2019 ; An et al., 2019 ; Hurtubise et al., 2020 ; Roye et al., 2019 ) about the epistemological ambiguity around the incentive status of control groups comprised of undergraduate research volunteers. Researchers long assumed that cognitively healthy university students assigned to the control condition would demonstrate their highest ability level by default.…”

“…Researchers long assumed that cognitively healthy university students assigned to the control condition would demonstrate their highest ability level by default. Some newer investigators began to question the validity of this unverified assumption, noting that research participants had been rewarded for their time but not for the quality of the data they produced (An et al, 2017;Powell et al, 2004;Roye et al, 2019;Russeler et al, 2008;Tan et al, 2002). Therefore, the magnitude of their incentive to fully comply with the instructions (i.e., appear impaired without being detected) does not match real-world malingerers who might be incentivized by 7-figure personal injury settlements (Dunn et al, 2003;Grant et al, 2020;Jelicic et al, 2011).…”

“…In other words, reminding real-world patients to provide valid data does not assure that outcome. Likewise, past studies showed that a variable proportion of student volunteers who were assigned to the control condition and asked to demonstrate their best ability in academic research settings failed PVTs ( An et al., 2012 ; DeRight & Jorgensen, 2015 ; Ross et al., 2016 ; Roye et al., 2019 ; Santos et al., 2014 ; Silk-Eglit et al., 2014 ). Ironically, participants assigned to the experimental malingering condition also occasionally demonstrated intact cognitive ability – in other words, they failed at failing ( Abeare et al., 2020 ).…”

“…However, based on the results of the original study, we hypothesized that the Yes/No Recognition and the RCFT FCR trials would be comparably sensitive to fluctuations in cognitive ability. Finally, we included a sample of students who were incentivized to perform well in order to address a separate limitation of the experimental malingering paradigm – variable motivation of research volunteers to demonstrate their maximal ability level as a control group ( An et al., 2012 ; Hurtubise et al., 2020 ; Lace et al., 2020 ; Roye et al., 2019 ). Given this added incentivizing, we expected the student controls in our sample to outperform Rai et al.’s (2019) control group.…”

Development of embedded performance validity indicators in the NIH Toolbox Cognitive Battery.

“…Keywords: NIH Toolbox Cognitive Battery, performance validity, mild traumatic brain injury, computerized neurocognitive test A fundamental assumption underlying cognitive assessment is that test scores accurately reflect the examinees' true ability level (Boone, 2013). This assumption is often violated, not only in forensic and clinical settings, but even among neurologically intact university students volunteering in research (An, Kaploun, Erdodi, & Abeare, 2017;Roye, Calamia, Bernstein, De Vito, & Hill, 2019). Performance validity tests (PVTs) have been developed to objectively measure the credibility of test results.…”