Standard Errors for National Trends in International Large-Scale Assessments in the Case of Cross-National Differential Item Functioning

Sachse, Karoline A.; Haag, Nicole

doi:10.1080/08957347.2017.1283315

Cited by 13 publications

(14 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Alternatively, it could be argued that marginal trend estimation also establishes a common metric across countries by placing the same identification constraints upon the item parameters for each country. In line with this reasoning, it has been shown that both original and marginal trend estimation provide unbiased estimates (see Sachse et al, 2016) but that marginal trends are more robust to the choice of link items (Carstensen, 2013) and that they are more efficient than original trends (Sachse et al, 2016;Sachse and Haag, 2017;Robitzsch and Lüdtke, 2019). Hence, we believe that marginal trend estimates also allow crossnational comparisons and that the estimation of marginal trends should receive more emphasis in large-scale assessment studies.…”

Section: Differences Between Original and Marginal Trend Estimates For Germanymentioning

confidence: 52%

“…Then, the country mean, based on the link items, may differ from the mean obtained from the international metric, which is computed from all items (i.e., link items and non-link items). As a consequence, the original trend estimate deviates from the marginal trend estimate, because the latter only takes link items into account (Monseur et al, 2008;Sachse and Haag, 2017).…”

Section: Computation Of Trend Estimates In Pisamentioning

confidence: 99%

See 1 more Smart Citation

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

et al. 2020

View full text Add to dashboard Cite

International large-scale assessments, such as the Program for International Student Assessment (PISA), are conducted to provide information on the effectiveness of education systems. In PISA, the target population of 15-year-old students is assessed every 3 years. Trends show whether competencies have changed in the countries between PISA cycles. In order to provide valid trend estimates, it is desirable to retain the same test conditions and statistical methods in all PISA cycles. In PISA 2015, however, the test mode changed from paper-based to computer-based tests, and the scaling method was changed. In this paper, we investigate the effects of these changes on trend estimation in PISA using German data from all PISA cycles (2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015). Our findings suggest that the change from paper-based to computer-based tests could have a severe impact on trend estimation but that the change of the scaling model did not substantially change the trend estimates.

show abstract

Section: Differences Between Original and Marginal Trend Estimates For Germanymentioning

confidence: 52%

Section: Computation Of Trend Estimates In Pisamentioning

confidence: 99%

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

et al. 2020

View full text Add to dashboard Cite

show abstract

“…This may be due to additional differences between our simulation and the real data: (a) The nonresponse mechanism in the real data might differ from the simulated missing data mechanism, (b) we simulated only two time points, (c) the amounts of missing data and the nonignorability decreased in the real data, whereas they increased in the simulation, or (d) the simulation did not include any not reached items (and the treatment of not reached items differed between the approaches). Further differences between the reported Irish trend in the official PISA reports and our analyses could be due to the fact that we calculated the marginal trend (Gebhardt & Adams, 2007), which is free from sources of error due to countryspecific differential item functioning that normally leads to additional errors in trend estimation when the original trend is estimated (Robitzsch & Lüdtke, 2018;Sachse & Haag, 2017). Certainly, the Irish data must contain many more idiosyncrasies that could potentially be responsible for the differences such as the fact that in 2009, school teachers rather than school inspectors administered the PISA tests (Perkins et al, 2012).…”

Section: Implications For Real Data Analysesmentioning

confidence: 93%

When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

Sachse

Mahler

Pohl

2019

Educational and Psychological Measurement

Self Cite

View full text Add to dashboard Cite

Mechanisms causing item nonresponses in large-scale assessments are often said to be nonignorable. Parameter estimates can be biased if nonignorable missing data mechanisms are not adequately modeled. In trend analyses, it is plausible for the missing data mechanism and the percentage of missing values to change over time. In this article, we investigated (a) the extent to which the missing data mechanism and the percentage of missing values changed over time in real large-scale assessment data, (b) how different approaches for dealing with missing data performed under such conditions, and (c) the practical implications for trend estimates. These issues are highly relevant because the conclusions hold for all kinds of group mean differences in large-scale assessments. In a reanalysis of PISA (Programme for International Student Assessment) data from 35 OECD countries, we found that missing data mechanisms and numbers of missing values varied considerably across time points, countries, and domains. In a simulation study, we generated data in which we allowed the missing data mechanism and the amount of missing data to change over time. We showed that the trend estimates were biased if differences in the missing-data mechanisms were not taken into account, in our case, when omissions were scored as wrong, when omissions were ignored, or when model-based approaches assuming a constant missing data mechanism over time were used. The results suggest that the most accurate estimates can be obtained from the application of multiple group models for nonignorable missing values when the amounts of missing data and the missing data mechanisms changed over time. In an empirical example, we furthermore showed that the large decline in PISA reading literacy in Ireland in 2009 was reduced when we estimated trends using missing data treatments that accounted for changes in missing data mechanisms.

show abstract

“…It has been emphasized in generalizability theory that the choice of items should also be included as part of statistical inference, like the sampling of persons [ 100 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 ]. The uncertainty with respect to items has been quantified as linking errors for trend estimates [ 109 , 110 , 111 ]. However, a similar error can also be computed for cross-sectional country means [ 34 , 112 , 113 ].…”

Section: Methodsmentioning

confidence: 99%

Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data

Robitzsch

2022

EJIHPE

View full text Add to dashboard Cite

In educational large-scale assessment (LSA) studies such as PISA, item response theory (IRT) scaling models summarize students’ performance on cognitive test items across countries. This article investigates the impact of different factors in model specifications for the PISA 2018 mathematics study. The diverse options of the model specification also firm under the labels multiverse analysis or specification curve analysis in the social sciences. In this article, we investigate the following five factors of model specification in the PISA scaling model for obtaining the two country distribution parameters; country means and country standard deviations: (1) the choice of the functional form of the IRT model, (2) the treatment of differential item functioning at the country level, (3) the treatment of missing item responses, (4) the impact of item selection in the PISA test, and (5) the impact of test position effects. In our multiverse analysis, it turned out that model uncertainty had almost the same impact on variability in the country means as sampling errors due to the sampling of students. Model uncertainty had an even larger impact than standard errors for country standard deviations. Overall, each of the five specification factors in the multiverse analysis had at least a moderate effect on either country means or standard deviations. In the discussion section, we critically evaluate the current practice of model specification decisions in LSA studies. It is argued that we would either prefer reporting the variability in model uncertainty or choosing a particular model specification that might provide the strategy that is most valid. It is emphasized that model fit should not play a role in selecting a scaling strategy for LSA applications.

show abstract

Standard Errors for National Trends in International Large-Scale Assessments in the Case of Cross-National Differential Item Functioning

Cited by 13 publications

References 31 publications

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification Curve Analysis for PISA 2018 Mathematics Data

Contact Info

Product

Resources

About