Low-stakes performance testing in Germany by the VERA assessment: analysis of the mode effects between computer-based testing and paper-pencil testing

Abstract

The German school system employs centrally organized performance assessments (some of which are called “VERA”) as a way of promoting lesson development. In recent years, several German federal states introduced a computer-based performance testing system which will replace the paper-pencil testing system in the future. Scores from computer-based testing are required to be equivalent to paper-pencil testing scores so that the new testing medium does not lead to disadvantages for students. Therefore, the current study aimed at investigating the size of the mode effect and the moderating impact of students’ gender, academic achievement and mainly spoken language in everyday life. In addition, the variance of the mode effect across tasks was investigated. The study was conducted in four German federal states in 2019 using a field experimental design. The test scores of 5140 eighth-graders from 165 schools in the subject German were analysed. The results of multi-level modelling revealed that students’ test scores in the computerized version of the VERA test were significantly lower than in the paper-pencil version. Students with a lower academic achievement were more disadvantaged by the VERA computerized test. The results were inconsistent regarding the interactions between testing mode and students’ gender and mainly spoken language in everyday life. The variance of the mode effect across tasks was high. Research into different subjects and in other federal states and countries under different testing conditions might yield further evidence about the generalizability of these results.

Remote Testing of Reading Comprehension in 8-Year-Old Children: Mode and Setting Effects

“…While children with higher reading abilities were hardly affected by changes in the administration conditions, for low ability children larger measurement biases were observed. Overall, these results replicate similar patterns that have been previously found for adolescents (Wagner et al, 2022) and young adults (Zinn et al, 2021). However, it must be emphasized that all effects found in the present study were rather small.…”

“…For example, 15-year-old students performed significantly worse on the Program for International Student Assessment (PISA) reading tests when administered on a computer as compared with paper ( Jerrim et al, 2018 ; Robitzsch et al, 2020 ). A similar pattern was also reported for mandatory state-wide student performance evaluations in Germany ( Wagner et al, 2022 ) that resulted in lower reading test performance on computerized test versions among eighth graders, particularly for low-achieving students. Other studies replicated these results for younger age groups, such as 10- to 13-year-old children ( Golan et al, 2018 ; Kerr & Symons, 2006 ; Støle et al, 2020 ).…”

Providing Individual Student Feedback at Scale for Mathematical Disciplines

“…Grades overall were reduced in the online version, but even more so for the students who were already weaker in the material being assessed. [13]. One can therefore postulate that a system that requires little learning from the individual students to become proficient users will be representative of the student's competence in the disciplinary material that is being assessed.…”