PISA 2015: how big is the ‘mode effect’ and what has been done about it?

Jerrim, John; Micklewright, John; Heine, Jörg-Henrik; Sälzer, Christine; McKeown, Caroline R.

doi:10.1080/03054985.2018.1430025

Cited by 48 publications

(52 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It is acknowledged that changing from paper-and-pencil to the new PC-and tabletbased administration could have substantial and unpredictable effects on student performance (APA 1986;Bennett et al 2008;Jerrim et al 2018;Mazzeo and von Davier 2014), which would have to be taken into consideration in analyzing and reporting the TIMSS 2019 results. These "mode effects" could vary systematically according to students' characteristics such as gender and their familiarity and confidence with using PCs and tablets (Bennett et al 2008;Cooper 2006;Gallagher et al 2002;Horkay et al 2006;Zhang et al 2016).…”

Section: Introductionmentioning

confidence: 99%

The TIMSS 2019 Item Equivalence Study: examining mode effects for computer-based assessment and implications for measuring trends

Fishbein

Martín

Mullis

et al. 2018

Large-scale Assess Educ

View full text Add to dashboard Cite

Background: TIMSS 2019 is the first assessment in the TIMSS transition to a computerbased assessment system, called eTIMSS. The TIMSS 2019 Item Equivalence Study was conducted in advance of the field test in 2017 to examine the potential for mode effects on the psychometric behavior of the TIMSS mathematics and science trend items induced by the change to computer-based administration. Methods: The study employed a counterbalanced, within-subjects design to investigate the potential for eTIMSS mode effects. Sample sizes for analysis included 16,894 fourth grade students from 24 countries and 9,164 eighth grade students from 11 countries. Following a review of the differences of the trend items in paper and digital formats, item statistics were examined item by item and aggregated by subject for paperTIMSS and eTIMSS. Then, the TIMSS scaling methods were applied to produce achievement scale scores for each mode. These were used to estimate the expected magnitude of the mode effects on student achievement. Results: The results of the study provide support that the mathematics and science constructs assessed by the trend items were mostly unaffected in the transition to eTIMSS at both grades. However, there was an overall mode effect, where items were more difficult for students in digital formats compared to paper. The effect was larger in mathematics than science. Conclusions: Because the trend items cannot be expected to be sufficiently equivalent across paperTIMSS and eTIMSS, it was concluded that modifications must be made to the usual item calibration model for TIMSS 2019 to measure trends. Each eTIMSS 2019 trend country will administer paper trend booklets to a nationally representative sample of students, in addition to the usual student sample, to provide a bridge between paperTIMSS and eTIMSS results.

show abstract

Section: Introductionmentioning

confidence: 99%

The TIMSS 2019 Item Equivalence Study: examining mode effects for computer-based assessment and implications for measuring trends

Fishbein

Martín

Mullis

et al. 2018

Large-scale Assess Educ

View full text Add to dashboard Cite

show abstract

“…The Programme for International Student Assessment (PISA) shifted to computer‐based testing in all domains in 2015 (OECD, ). Following the publication of the results, some authors found indications that cross‐sectional comparisons as well as the comparability of estimates to previous assessments (i.e., trend estimates) were probably influenced by the change from paper and pencil to computer tests ( mode effects ; e.g., Jerrim, Micklewright, Heine & Sälzer, ; Klieme, ; Komatsu & Rappleye, ; Robitzsch et al, ; see also Jerrim, ), calling into question whether potential mode effects were effectively handled in the procedures applied to the PISA 2015 cognitive measures.…”

mentioning

confidence: 99%

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Kroehne

Buerger

Hahnel

et al. 2019

Educational Measurement

View full text Add to dashboard Cite

For many years, reading comprehension in the Programme for International Student Assessment (PISA) was measured via paper‐based assessment (PBA). In the 2015 cycle, computer‐based assessment (CBA) was introduced, raising the question of whether central equivalence criteria required for a valid interpretation of the results are fulfilled. As an extension of the PISA 2012 main study in Germany, a random subsample of two intact PISA reading clusters, either computerized or paper‐based, was assessed using a random group design with an additional within‐subject variation. The results are in line with the hypothesis of construct equivalence. That is, the latent cross‐mode correlation of PISA reading comprehension was not significantly different from the expected correlation between the two clusters. Significant mode effects on item difficulties were observed for a small number of items only. Interindividual differences found in mode effects were negatively correlated with reading comprehension, but were not predicted by basic computer skills or gender. Further differences between modes were found with respect to the number of missing values.

show abstract

“…However, reanalyses of the field trial data collected in Germany found national level mode effects where items administered on a computer were more difficult compared to a paper and pen administration (Robitzsch et al 2017). Jerrim et al (2018) concluded (based on a reanalyses of field trial data from Germany, Sweden, and Ireland) that the measures taken during the 2015 cycle have reduced, but not completely resolved, the impact of mode effects.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

“…Robitzsch and colleagues (2017) came to this conclusion by re-analysing the PISA data and estimating the mean German performances comparing several calibration designs. Another recent study conducted by Jerrim et al (2018) also questions the comparability of PBA and CBA results. Both studies are among others based upon a reanalysis of the 2014 PISA field trial study data available for some countries.…”

Section: Introductionmentioning

confidence: 99%

Differential Item Functioning in PISA Due to Mode Effects

Feskens

Fox

Zwitser

2019

Theoretical and Practical Advances in Computer-Based Educational Measurement

View full text Add to dashboard Cite

One of the most important goals of the Programme for International Student Assessment (PISA) is assessing national changes in educational performance over time. These so-called trend results inform policy makers about the development of ability of 15-year-old students within a specific country. The validity of those trend results prescribes invariant test conditions. In the 2015 PISA survey, several alterations to the test administration were implemented, including a switch from paper-based assessments to computer-based assessments for most countries (OECD 2016a). This alteration of the assessment mode is examined by evaluating if the items used to assess trends are subject to differential item functioning across PISA surveys (2012 vs. 2015). Furthermore, the impact on the trend results due to the change in assessment mode of the Netherlands is assessed. The results show that the decrease reported for mathematics in the Netherlands is smaller when results are based upon a separate national calibration.

show abstract

PISA 2015: how big is the ‘mode effect’ and what has been done about it?

Cited by 48 publications

References 16 publications

The TIMSS 2019 Item Equivalence Study: examining mode effects for computer-based assessment and implications for measuring trends

The TIMSS 2019 Item Equivalence Study: examining mode effects for computer-based assessment and implications for measuring trends

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Differential Item Functioning in PISA Due to Mode Effects

Contact Info

Product

Resources

About