Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

Abstract

International large-scale assessments, such as the Program for International Student Assessment (PISA), are conducted to provide information on the effectiveness of education systems. In PISA, the target population of 15-year-old students is assessed every 3 years. Trends show whether competencies have changed in the countries between PISA cycles. In order to provide valid trend estimates, it is desirable to retain the same test conditions and statistical methods in all PISA cycles. In PISA 2015, however, the test mode changed from paper-based to computer-based tests, and the scaling method was changed. In this paper, we investigate the effects of these changes on trend estimation in PISA using German data from all PISA cycles (2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015). Our findings suggest that the change from paper-based to computer-based tests could have a severe impact on trend estimation but that the change of the scaling model did not substantially change the trend estimates.

The inattentive on-screen reading: Reading medium affects attention and reading comprehension under time pressure

“…For example, taking exams on screen could prevent students from fully demonstrating their knowledge and skills, because they may struggle to adjust their attentional focus to their full potential. Recent studies analyzing the change of the mode (from print to computer) in the PISA tests support this conclusion, as indicated by lower scores from the computer-based tests ( Jerrim et al, 2018 ; Robitzsch, Lüdtke, Goldhammer, Kroehne, & Köller, 2020 ). Moreover, mindwandering has demonstrated to exert a negative impact on academic performance ( Hollis & Was, 2016 ; Seli, Wammes, Risko, & Smilek, 2016 ).…”

“…Furthermore, as the negative effect of on-screen reading especially arises when reading under time constraints ( Ackerman & Lauterman, 2012 ; Delgado et al, 2018 ), special attention should be taken when conducting timed exams on screens. Recent studies analysing data from the Program for International Student Assessment (PISA) have found that the change in presentation mode from print to computerized tests in PISA 2015 had a detrimental impact on students’ PISA test scores in Germany, Sweden and Ireland ( Jerrim, Micklewright, Heine, Salzer, & McKeown, 2018 ; Robitzsch, Lü;dtke, Goldhammer, Kroehne, & Köller, 2020 ). In a time when online education is becoming ubiquitous due to worldwide lockdowns associated to the COVID-19 pandemic, it is urgent to understand the underlying mechanisms of the medium effect as a first step to minimize the negative educational effects of on-screen reading.…”

Robust Haebara Linking for Many Groups: Performance in the Case of Uniform DIF

“…For South Korea, there are four items with large negative DIF effects (a relative advantage) and no items with large positive DIF effects (a relative disadvantage) that are most strongly down-weighted (see [10]). Hence, it can be concluded the choice of a particular linking method has the potential to impact the ranking of countries in PISA (see also [48,49]).…”

$L_p$ Loss Functions in Invariance Alignment and Haberman Linking

“…The linking of multiple groups using other linking functions has been treated in [84,[87][88][89]. In contrast to the IA method, HL has only been scarcely applied [90][91][92][93][94][95][96][97][98].…”