Herausforderungen bei der Schätzung von Trends in Schulleistungsstudien

Robitzsch, Alexander; Lüdtke, Oliver; Köller, Olaf; Kröhne, Ulf; Goldhammer, Frank; Heine, Jörg-Henrik

doi:10.1026/0012-1924/a000177

Cited by 21 publications

(27 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…At first sight, the finding of longer and more relevant responses seems contradictory to previous findings that tasks become more difficult on the computer (Kolen & Brennan, 2014;Robitzsch et al, 2017). However, a more detailed look at the variation of the effects across items puts this in a different light, in line with the inconsistent findings of mode effect studies.…”

Section: Discussioncontrasting

confidence: 72%

See 1 more Smart Citation

Unattended consequences: how text responses alter alongside PISA’s mode change from 2012 to 2015

et al. 2018

Self Cite

View full text Add to dashboard Cite

In 2015, the Programme for International Student Assessment (PISA) introduced multiple changes in its study design, the most extensive being the transition from paper-to computer-based assessment. We investigated the differences between German students' text responses to eight reading items from the paper-based study in 2012 to text responses to the same items from the computer-based study in 2015. Two response featuresinformation quantity and relevance proportionwere extracted by natural language processing techniques because they are crucial indicators for the response process. Showcasing potential differential relationships, we additionally examined gender differences. Modelling effects of the round of assessment, gender, and response correctness on the response features, we analysed responses from 15-year-olds and ninth-graders in Germany. Results revealed differences in the text responses between the rounds of assessment in that students included more information overall in 2015, and the proportions of relevance varied substantially across items. As the study investigated the mode change in PISA's natural (not experimental) setting, the differences could mirror cohort trends or design changes. However, with the evidence reported, we conclude that the differences could indicate mode effects. KEYWORDS Computer-based assessment; paper-based assessment; open-ended text responses; mode effect; automatic processing CONTACT Fabian Zehner

show abstract

Section: Discussioncontrasting

confidence: 72%

“…The OECD acknowledges the possibility of potential small to medium mode effects for PISA 2015 . For Germany indeed, Robitzsch et al (2017) found a small mode effect. In line with the literature (e.g.…”

Section: Introductionmentioning

confidence: 91%

Unattended consequences: how text responses alter alongside PISA’s mode change from 2012 to 2015

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…An exhaustive review of previous research is beyond the scope of this article. However, as Robitzsch et al.’s () results give reason to assume domain‐specific results, we present a more detailed review of previous results on the comparability of reading assessments.…”

Section: Previous Results On Mode Effects Of Reading Assessmentmentioning

confidence: 88%

“…The Programme for International Student Assessment (PISA) shifted to computer‐based testing in all domains in 2015 (OECD, ). Following the publication of the results, some authors found indications that cross‐sectional comparisons as well as the comparability of estimates to previous assessments (i.e., trend estimates) were probably influenced by the change from paper and pencil to computer tests ( mode effects ; e.g., Jerrim, Micklewright, Heine & Sälzer, ; Klieme, ; Komatsu & Rappleye, ; Robitzsch et al, ; see also Jerrim, ), calling into question whether potential mode effects were effectively handled in the procedures applied to the PISA 2015 cognitive measures.…”

mentioning

confidence: 99%

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Kroehne

Buerger

Hahnel

et al. 2019

Educational Measurement

Self Cite

View full text Add to dashboard Cite

For many years, reading comprehension in the Programme for International Student Assessment (PISA) was measured via paper‐based assessment (PBA). In the 2015 cycle, computer‐based assessment (CBA) was introduced, raising the question of whether central equivalence criteria required for a valid interpretation of the results are fulfilled. As an extension of the PISA 2012 main study in Germany, a random subsample of two intact PISA reading clusters, either computerized or paper‐based, was assessed using a random group design with an additional within‐subject variation. The results are in line with the hypothesis of construct equivalence. That is, the latent cross‐mode correlation of PISA reading comprehension was not significantly different from the expected correlation between the two clusters. Significant mode effects on item difficulties were observed for a small number of items only. Interindividual differences found in mode effects were negatively correlated with reading comprehension, but were not predicted by basic computer skills or gender. Further differences between modes were found with respect to the number of missing values.

show abstract

“…Since PISA introduced CBA in 2015 [18], some authors have questioned the comparability of CBA and PBA in PISA, in particular, concerning trend estimates (e.g., [19,20]). Although a detailed review of the current literature regarding mode effects is beyond the scope of this paper (see, e.g., [8]), we aim at providing a method for investigating response time differences between modes, which is currently missing.…”

Section: Invariance Of the Response Processes Between Gender And Modementioning

confidence: 99%

Invariance of the Response Processes Between Gender and Modes in an Assessment of Reading

Kroehne

Hahnel

Goldhammer

2019

Front. Appl. Math. Stat.

Self Cite

View full text Add to dashboard Cite

In this paper, we developed a method to extract item-level response times from log data that are available in computer-based assessments (CBA) and paper-based assessments (PBA) with digital pens. Based on response times that were extracted using only time differences between responses, we used the bivariate generalized linear IRT model framework (B-GLIRT, [1]) to investigate response times as indicators for response processes. A parameterization that includes an interaction between the latent speed factor and the latent ability factor in the cross-relation function was found to fit the data best in CBA and PBA. Data were collected with a within-subject design in a national add-on study to PISA 2012 administering two clusters of PISA 2009 reading units. After investigating the invariance of the measurement models for ability and speed between boys and girls, we found the expected gender effect in reading ability to coincide with a gender effect in speed in CBA. Taking this result as indication for the validity of the time measures extracted from time differences between responses, we analyzed the PBA data and found the same gender effects for ability and speed. Analyzing PBA and CBA data together we identified the ability mode effect as the latent difference between reading measured in CBA and PBA. Similar to the gender effect the mode effect in ability was observed together with a difference in the latent speed between modes. However, while the relationship between speed and ability is identical for boys and girls we found hints for mode differences in the estimated parameters of the cross-relation function used in the B-GLIRT model.Technology-based assessments offer the possibility to collect additional log data, including response times as the amount of time test-takers spend responding to particular questions or tasks. Primarily, response times provide information about test-takers' speed, which-if not considered-might confound comparisons of test-takers' ability [2] due to inter-individual differences in the speed-ability compromise [3]. Response times can allow inferences about response processes [4,5]. Although at the individual level response times are influenced by multiple factors [6], the comparison of characteristics at the group level, such as the between-person relationship of speed

show abstract

Herausforderungen bei der Schätzung von Trends in Schulleistungsstudien

Cited by 21 publications

References 14 publications

Unattended consequences: how text responses alter alongside PISA’s mode change from 2012 to 2015

Unattended consequences: how text responses alter alongside PISA’s mode change from 2012 to 2015

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Invariance of the Response Processes Between Gender and Modes in an Assessment of Reading

Contact Info

Product

Resources

About