“…Future research should investigate whether other methods, for example, the methods suggested by Oliveri and von Davier () or by Fox and Verhagen (), could be considered alternatives that are more accurate, because our study's results provide no generalization to these methods. Further, it would be interesting to determine the nature and stability of cross‐national DIF over time in different ILSAs.…”
Section: Discussionmentioning
confidence: 92%
“…This study compared the performances of three different linking methods for national trend estimation in international large‐scale assessments under various simulated conditions with different amounts of cross‐national DIF. A systematic investigation of cross‐national DIF in linking was essential because many methods applied to educational large‐scale assessments rely on cross‐national measurement invariance, and recent studies have repeatedly demonstrated that this assumption may not hold and that alternative methods should be considered (Carstensen et al., ; Fox & Verhagen, ; Gebhardt & Adams, ; Grisay & Monseur, ; Kankaraš & Moors, ; Kreiner & Christensen, ; Oliveri & von Davier, ; Rutkowski & Svetina, ). In particular, we aimed to compare the operational practice used by PISA until 2015 to two alternative methods for national trend estimation in order to identify the amounts of DIF for which each method performed best such that practitioners could get an indication of when (i.e., at what amount of DIF) it would become beneficial to switch methods.…”
Section: Discussionmentioning
confidence: 99%
“…Researchers have developed methods that allow for cross‐national variation in item parameters (e.g., Fox & Verhagen, ) to prevent trend estimation from being affected by cross‐national DIF. Moreover, Oliveri and von Davier (, ) proposed that all data be concurrently calibrated by using multiple group models and that misfitting international item parameters be replaced by item parameters that were based on national data for constructing the international scale, which resulted in an improved model‐data fit. However, all these methods were not routinely applied to the aforementioned educational large‐scale assessments until 2015 (in PISA, changes will be made from the 2015 cycle on).…”
Trend estimation in international comparative large-scale assessments relies on measurement invariance between countries. However, cross-national differential item functioning (DIF) has been repeatedly documented. We ran a simulation study using national item parameters, which required trends to be computed separately for each country, to compare trend estimation performances to two linking methods employing international item parameters across several conditions. The trend estimates based on the national item parameters were more accurate than the trend estimates based on the international item parameters when cross-national DIF was present. Moreover, the use of fixed common item parameter calibrations led to biased trend estimates. The detection and elimination of DIF can reduce this bias but is also likely to increase the total error.
152
“…Future research should investigate whether other methods, for example, the methods suggested by Oliveri and von Davier () or by Fox and Verhagen (), could be considered alternatives that are more accurate, because our study's results provide no generalization to these methods. Further, it would be interesting to determine the nature and stability of cross‐national DIF over time in different ILSAs.…”
Section: Discussionmentioning
confidence: 92%
“…This study compared the performances of three different linking methods for national trend estimation in international large‐scale assessments under various simulated conditions with different amounts of cross‐national DIF. A systematic investigation of cross‐national DIF in linking was essential because many methods applied to educational large‐scale assessments rely on cross‐national measurement invariance, and recent studies have repeatedly demonstrated that this assumption may not hold and that alternative methods should be considered (Carstensen et al., ; Fox & Verhagen, ; Gebhardt & Adams, ; Grisay & Monseur, ; Kankaraš & Moors, ; Kreiner & Christensen, ; Oliveri & von Davier, ; Rutkowski & Svetina, ). In particular, we aimed to compare the operational practice used by PISA until 2015 to two alternative methods for national trend estimation in order to identify the amounts of DIF for which each method performed best such that practitioners could get an indication of when (i.e., at what amount of DIF) it would become beneficial to switch methods.…”
Section: Discussionmentioning
confidence: 99%
“…Researchers have developed methods that allow for cross‐national variation in item parameters (e.g., Fox & Verhagen, ) to prevent trend estimation from being affected by cross‐national DIF. Moreover, Oliveri and von Davier (, ) proposed that all data be concurrently calibrated by using multiple group models and that misfitting international item parameters be replaced by item parameters that were based on national data for constructing the international scale, which resulted in an improved model‐data fit. However, all these methods were not routinely applied to the aforementioned educational large‐scale assessments until 2015 (in PISA, changes will be made from the 2015 cycle on).…”
Trend estimation in international comparative large-scale assessments relies on measurement invariance between countries. However, cross-national differential item functioning (DIF) has been repeatedly documented. We ran a simulation study using national item parameters, which required trends to be computed separately for each country, to compare trend estimation performances to two linking methods employing international item parameters across several conditions. The trend estimates based on the national item parameters were more accurate than the trend estimates based on the international item parameters when cross-national DIF was present. Moreover, the use of fixed common item parameter calibrations led to biased trend estimates. The detection and elimination of DIF can reduce this bias but is also likely to increase the total error.
152
“…More comparable results can be achieved with larger numbers of narrower scales, multidimensional models, or models that are simpler in the main but incorporate specific departures by nation, culture, language group, or latent classes that may be associated with curricula (e.g., Oliveri & von Davier, 2014). The resulting story is more complicated, but it is because the patterns in the data are more complicated.…”
Section: Challenge 3: How Do We Think About Constructs?mentioning
Validity is the sine qua non of properties of educational assessment. While a theory of validity and a practical framework for validation has emerged over the past decades, most of the discussion has addressed familiar forms of assessment and psychological framings. Advances in digital technologies and in cognitive and social psychology have expanded the range of purposes, targets of inference, contexts of use, forms of activity, and sources of evidence we now see in educational assessment. This article discusses some of these developments and how concepts and representations that are employed to design and use assessments, hence to frame validity arguments, can be extended accordingly. Ideas are illustrated with a variety of examples, with an emphasis on assessment in higher education.
“…Such analyses may be helpful in evaluating item invariance, which could indicate whether the items are appropriate for the new population. Oliveri and von Davier () suggest as follows: . .…”
Section: Domain Analysis Evaluation and Generalization Componentsmentioning
In this paper, we first examine the challenges of score comparability associated with the use of assessments that are exported. By exported assessments, we mean assessments that are developed for domestic use and are then administered in other countries in either the same or a different language. Second, we provide suggestions to better support their valid and fair use. We illustrate these issues in the context of higher education assessments that are designed to serve different purposes—inform admissions decisions and assess student learning outcomes within one country (e.g., the United States) that are later used in other countries. In higher education, the use of exported assessments is on the rise due to increases in globalization, student mobility, and cross‐national comparisons of student achievement. An increase in the use of exported assessments leads to more diverse test‐taker populations and requires special attention due to possible sources of construct‐irrelevant variance, which may threaten the score‐based inferences made for various populations. Irrelevant sources of variance may emerge due to differences in opportunity to learn, curricular exposure, and lack of familiarity with the cultural references used in the assessments that are exported.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
