Unproctored internet-based device-type effects on test scores: The role of working memory

The testing environment as an explanation for unproctored internet‐based testing device‐type effects

“…Despite approximating the operational context to a greater degree than prior nonoperational studies, there is an evident mismatch between the typical operational context and the present study. Specifically, in addition to the testing environment, Arthur and colleagues (Arthur et al., 2014; Arthur, et al., 2018; Arthur & Traylor, 2019) posited two additional explanations for why UIT device‐type effects have been found only in operational contexts: (a) the high‐stakes nature of employment‐related testing, and (b) applicants' ability to choose the device on which UITs are completed. That is, the testing stakes for participants in operational studies (i.e., job applicants) are unequivocally high because applicants are vying for a limited number of job openings.…”

“…GMA was operationalized as scores on a timed, 60‐item (36 verbal and 24 quantitative), 4‐alternative, multiple‐choice test (GMA 60 ; Arthur, 2017; Arthur, et al., 2018). The test was optimized for mobile devices, and participants were allotted 10 min to complete the test.…”

“…Despite these and other contrary findings, there are several, noteworthy differences in study type and context between the studies that have and have not obtained UIT device‐type effects which might account for the inconsistent findings (Arthur, et al., 2018; Arthur & Traylor, 2019). Operational studies are, of course, characterized by high testing stakes whereby participants (i.e., applicants) compete for a limited number of prized job openings.…”

“…Despite these seemingly straightforward findings, Arthur et al. (2018) noted an oddity about this literature; specifically, whereas UIT device‐type effects have been consistently demonstrated by operational studies (e.g., Arthur et al., 2014; Wood et al., 2015), those in nonoperational settings such as research laboratories have consistently demonstrated no such effects (e.g., Arthur, et al., 2018; Brown & Grossenbacher, 2017; Parker & Meade, 2015). That is, for this latter group of studies, there were no differences in cognitive test scores between those who completed the test on ‘mobile’ versus ‘nonmobile’ devices.…”

“…Prior research (Arthur et al., 2014; Arthur, et al., 2018; Arthur & Traylor, 2019) has identified three plausible explanations for the preceding discrepancy: (1) differences in testing stakes (i.e., high‐ vs. low‐stakes), (2) the formation of device‐type conditions (i.e., self‐selection vs. random assignment to device type), and (3) the extent of environmental distractions (i.e., completing assessments in distracting vs. nondistracting environments). The objective of the present study is to examine the viability of the latter—the testing environment—as an explanation for these discrepant findings (Arthur et al., 2018).…”