Mode Effects in College Admissions Testing and Differential Speededness as a Possible Explanation

Abstract

As testing programs transition from paper to online testing, they must study mode comparability to support the exchangeability of scores from different testing modes. To that end, a series of three mode comparability studies was conducted during the 2019–2020 academic year with examinees randomly assigned to take the ACT college admissions exam on paper or online. Unlike many prior mode comparability studies, examinees testing online earned higher scores on average than examinees testing on paper, especially on the English, reading, and writing tests. Much of the observed mode effects were accounted for by items near the ends of the tests. Subsequent examination of item omit rates, differential item functioning by mode, and change‐point analyses supported the notion that the observed mode effects were related to differential speededness. Specifically, examinees testing online were slightly less speeded, and were therefore able to perform better on items near the ends of the tests. Practical implications for linking across modes, test design, and test administration conditions are discussed.

Remote Testing of Reading Comprehension in 8-Year-Old Children: Mode and Setting Effects

“…Others proposed differences in test-taking behavior as a potential explanation. For example, test-takers tend to take less time on the computer and finish tests quicker, while showing higher guessing behavior, particularly among low-performance students ( Karay et al, 2015 ; Singer et al, 2019 ); albeit also opposite results were sometimes observed ( Steedle et al, 2022 ). An experimental study that independently varied the presentation medium (paper versus computer) of the reading text and the test items suggested that the mode effect in reading comprehension is primarily a result of digital reading rather than a media-induced testing effect ( Ben-Yehudah & Eshet-Alkalai, 2021 ).…”

“…Particularly, for cognitive measurements in educational contexts computerized proficiency testing has become the de facto standard in many applied areas. For example, many high-stake college admission or proficiency certification procedures adopt computerized testing formats (e.g., Hurtz & Weiner, 2022;Steedle et al, 2022) because they allow for better standardization of test instructions, item presentations, and response coding, thus, leading to less error-prone and fairer measurements. Even many educational large-scale studies have recently switched to computers as their preferred medium of assessment that also allows for administering innovative item formats (e.g., simulation-based items) and collecting ancillary information (e.g., process data) to capture novel constructs more precisely (see von Davier et al, 2019).In recent years, psychological assessments have faced another significant shift.…”