A Meta-Analysis of Test Scores in Proctored and Unproctored Ability Assessments

Abstract

Abstract. Unproctored, web-based assessments are frequently compromised by a lack of control over the participants’ test-taking behavior. It is likely that participants cheat if personal consequences are high. This meta-analysis summarizes findings on context effects in unproctored and proctored ability assessments and examines mean score differences and correlations between both assessment contexts. As potential moderators, we consider (a) the perceived consequences of the assessment, (b) countermeasures against cheating, (c) the susceptibility to cheating of the measure itself, and (d) the use of different test media. For standardized mean differences, a three-level random-effects meta-analysis based on 109 effect sizes from 49 studies (total N = 100,434) identified a pooled effect of Δ = 0.20, 95% CI [0.10, 0.31], indicating higher scores in unproctored assessments. Moderator analyses revealed significantly smaller effects for measures that are difficult to research on the Internet. These results demonstrate that unproctored ability assessments are biased by cheating. Unproctored assessments may be most suitable for tasks that are difficult to search on the Internet.

The Longitudinal Measurement of Reasoning Abilities in Students With Special Educational Needs

“…Although great care was taken to make the two assessments as comparable as possible, systematic mode effects cannot be ruled out (cf. Steger et al, 2018; Kroehne et al, 2019). For example, responding on the computer might be more difficult as compared to dealing with paper-based booklets because of differences in computer familiarity.…”

Internet‐Based Proctored Assessment: Security and Fairness Issues

“…Bloemers et al (2016) found that test takers occasionally violated UIT behavior agreements by receiving assistance from others, searching the Internet, and copying test content. Steger et al (2020), in their meta-analysis of 49 studies with over 100,000 test takers, found an effect size of 0.20 standard deviation units favoring those testing in an UIT environment. Despite this finding, they found that effect sizes were reduced to near zero when (a) a test had strict time limits, (b) content was not Internet searchable, and (c) a lockdown browser was employed.…”

“…Despite these features, researchers have raised questions regarding score integrity and content security (Tippins et al, 2006). Studies evaluating UIT for high-stakes cognitive ability tests have generally found that score distributions tend to be higher than in proctored conditions (Arthur et al, 2010;Bloemers, Oud, & van Dam, 2016;Reynolds, Wasko, Sinar, Raymark, & Jones, 2009;Steger, Schroeders, & Gnambs, 2020). Bloemers et al (2016) found that test takers occasionally violated UIT behavior agreements by receiving assistance from others, searching the Internet, and copying test content.…”

“…Despite this finding, they found that effect sizes were reduced to near zero when (a) a test had strict time limits, (b) content was not Internet searchable, and (c) a lockdown browser was employed. Despite these safeguards, the overall effect sizes found by Steger et al (2020) provide reasons for concern. Programs striving to develop an UIT programs need to carefully evaluate their program design.…”

Percepciones sobre amor, compromiso, fidelidad y pareja en jóvenes universitarios de Quito

“…La encuesta fue aplicada en línea. Si bien algunos estudios cuestionan los aspectos negativos de las evaluaciones en línea (Steger, Schroeders, y Gnambs, 2020), se consideró que en este caso era el tipo de aplicación óptima por la facilidad de acceso a la población y la tranquilidad requerida por los participantes para contestar a las preguntas en relación al alto número de ítems. Se consideró que se obtendrían respuestas confiables por las siguientes razones: (1) no se trataba de una evaluación de habilidades específicas, sino de actitudes personales frente a los temas, (2) la participación era enteramente voluntaria, lo que determinaba que el participante podía no finalizar su encuesta si así lo deseaba, y (3) los participantes no obtenían ninguna retribución o sanción por realizarla o no.…”