A Comparison of Linking Methods for Estimating National Trends in International Comparative Large‐Scale Assessments in the Presence of Cross‐National DIF

Abstract

Trend estimation in international comparative large-scale assessments relies on measurement invariance between countries. However, cross-national differential item functioning (DIF) has been repeatedly documented. We ran a simulation study using national item parameters, which required trends to be computed separately for each country, to compare trend estimation performances to two linking methods employing international item parameters across several conditions. The trend estimates based on the national item parameters were more accurate than the trend estimates based on the international item parameters when cross-national DIF was present. Moreover, the use of fixed common item parameter calibrations led to biased trend estimates. The detection and elimination of DIF can reduce this bias but is also likely to increase the total error. 152

Robust Haebara Linking for Many Groups: Performance in the Case of Uniform DIF

“…It should be emphasized that we did not investigate the computation of standard errors in our linking approach. There is ample literature that derives standard error formulas for linking due to sampling of persons (e.g., [44,50,[63][64][65][66][67]) Alternatively, variability in estimated group means due to the selection of items has been studied as linking errors in the literature [47,[68][69][70][71][72]. In future research, it would be interesting to accompany robust Haebara linking with error components that reflect these sources of uncertainty [24,64,73].…”

Detecting cross-cultural differential item functioning for increasing validity: An example from the American board of family medicine in-training examination

“…[11] Beyond gender and ethnicity DIF, cross-cultural DIF (cc-DIF) has increasingly received attention from researchers in a variety of fields. [8,[12][13][14][15][16] In some international comparison studies, the terms cc-DIF and cross-national DIF are used interchangeably. Culture refers to a broader category, beyond location and ethnicity, to include any group of people who share common lifestyle characteristics that are passed on to members of the group, such as socioeconomic status and religion.…”

“…[17] Researchers have shown that not all items exhibit measurement invariance across cultures. [13,15,18] Measurement invariance refers to measurement operations yielding measures of the same attribute that does not vary across different conditions and situations. [19] When an item functions differently in a specific culture in comparison with the item functioning across other cultures, cc-DIF appears.…”

“…[19] When an item functions differently in a specific culture in comparison with the item functioning across other cultures, cc-DIF appears. [15] The four most likely reasons for explaining cc-DIF are (1) language translation, [20] (2) cultural specificities, [11,12] (3) differential curriculum coverage, [21] and (4) differential content familiarity. [22] DIF across cultures caused by language translation and differential curriculum coverage are generally easier to identify and correct, whereas DIF introduced by cultural specificities and differential content familiarity can be implicit and require more attention.…”

“…Further, cc-DIF has been found to negatively influence both the accuracy of scores and the inferences made about them. [8,12,15] Therefore, investigating cc-DIF can help reduce item bias and increase validity. [2,23] Given the importance of measurement invariance and test validity, the purpose of this study was to draw attention to the issue of cc-DIF and demonstrate how to detect cc-DIF using a real dataset from a medical certification board located in the United States (US).…”