Identifying Sources of Differential Item and Bundle Functioning on Translated Achievement Tests: A Confirmatory Analysis

Abstract

Increasingly, tests are being translated and adapted into different languages. Differential item functioning (DIF) analyses are often used to identify non‐equivalent items across language groups. However, few studies have focused on understanding why some translated items produce DIF. The purpose of the current study is to identify sources of differential item and bundle functioning on translated achievement tests using substantive and statistical analyses. A substantive analysis of existing DIF items was conducted by an 11‐member committee of testing specialists. In their review, four sources of translation DIF were identified. Two certified translators used these four sources to categorize a new set of DIF items from Grade 6 and 9 Mathematics and Social Studies Achievement Tests. Each item was associated with a specific source of translation DIF and each item was anticipated to favor a specific group of examinees. Then, a statistical analysis was conducted on the items in each category using SIBTEST. The translators sorted the mathematics DIF items into three sources, and they correctly predicted the group that would be favored for seven of the eight items or bundles of items across two grade levels. The translators sorted the social studies DIF items into four sources, and they correctly predicted the group that would be favored for eight of the 13 items or bundles of items across two grade levels. The majority of items in mathematics and social studies were associated with differences in the words, expressions, or sentence structure of items that are not inherent to the language and/or culture. By combining substantive and statistical DIF analyses, researchers can study the sources of DIF and create a body of confirmed DIF hypotheses that may be used to develop guidelines and test construction principles for reducing DIF on translated tests.

Detecting Multidimensional Differential Item Functioning with the Multiple Indicators Multiple Causes Model, the Item Response Theory Likelihood Ratio Test, and Logistic Regression

“…Also, it should be noted that the identification of DIF items may not imply a significant bias in the items. Therefore, the sources of DIF should be identified to ensure that DIF items do not lead to unfairness (e.g., Gierl and Khaliq, 2001;Stark et al, 2004;Chernyshenko et al, 2007). If the presence of DIF is related to unintended content or property in the item, then the item can be considered unfair (Penfield and Camilli, 2007).…”

International Education Studies: Increasing Their Linguistic Comparability by Developing Judgmental Reviews

“…As revealed by this study and several other studies (e.g., [7,9,15]), differences in grammatical form and structure and differences in meanings and connotations are a common potential source of bias in international education studies. Moreover, contrary to the view that seems to have prevailed in PISA 2000, that language-specific differences are repairable [30], often not much can be done to correct these differences and to eliminate the bias [13,40].…”

“…During the first PISA round, reviewers were still encouraged to look also for language-specific differences [30], but in more recent studies, no mention has been made of them (e.g., [31]). However, there is ample research suggesting that differences between languages often lead to bias (e.g., [7,9,15]). Therefore, it is important to include also these differences in the definition of translation problems (see also [13]).…”

“…A good deal of information already exists on typical sources of language-related bias, which can be used when generating items. However, it is good to continue collecting more such data during each PISA round and also from other sources (see also [7,15]). For example, from this study one can see that because of language-specific differences in form, items which are based on and ask about the exact wording of an expression (like Item 1) are always risky (see also [11]).…”

“…Research also shows that often the reviews have been unable to identify sources of DIF consistently and accurately. For example, it has been detected that reviewers have frequently been unable to correctly predict DIF and the direction of DIF (e.g., [14][15][16]). Ercikan et al [11] found that reviewers' judgments as to which items are biased and why are frequently incorrect and, therefore, need to be verified by checking them against what the testees themselves think.…”