Application of Think Aloud Protocols for Examining and Confirming Sources of Differential Item Functioning Identified by Expert Reviews

Abstract

This paper demonstrates and discusses the use of think aloud protocols (TAPs) as an approach for examining and confirming sources of differential item functioning (DIF). The TAPs are used to investigate to what extent surface characteristics of the items that are identified by expert reviews as sources of DIF are supported by empirical evidence from examinee thinking processes in the English and French versions of a Canadian national assessment. In this research, the TAPs confirmed sources of DIF identified by expert reviews for 10 out of 20 DIF items. The moderate agreement between TAPs and expert reviews indicates that evidence from expert reviews cannot be considered sufficient in deciding whether DIF items are biased and such judgments need to include evidence from examinee thinking processes.

International Education Studies: Increasing Their Linguistic Comparability by Developing Judgmental Reviews

“…This information provided indirect data (or clues) concerning the students' cognitive processes (e.g., what factors may have affected their answering; cf. [11]), thereby making it possible to make more informed guesses about the sources of DIF. The latter document also revealed whether any language-related explanations were suggested by the reviewer for the poor functioning of the items.…”

“…For example, it has been detected that reviewers have frequently been unable to correctly predict DIF and the direction of DIF (e.g., [14][15][16]). Ercikan et al [11] found that reviewers' judgments as to which items are biased and why are frequently incorrect and, therefore, need to be verified by checking them against what the testees themselves think.…”

“…Language-related differences refer to differences between languages and translation errors and can be roughly classified as in Table 1 (cf. [7][8][9][10][11][12][13]). Of the six language-related sources of bias, the first four typically derive from differences between languages whereas the last two are concerned with faulty or inappropriate translation.…”

Investigating Linguistic Sources of Differential Item Functioning Using Expert Think-Aloud Protocols in Science Achievement Tests

“…In the context of identifying sources of DIF in English and French versions of a Canadian assessment, think-aloud protocols have been proposed recently as a new approach (Ercikan et al, 2010). These researchers examined whether examinees' think-aloud protocols confirmed the linguistic differences identified by expert reviewers as sources of DIF.…”

“…Expert (or judgmental) review -reviews of items by individuals who are knowledgeable about student learning and may have cultural or linguistic expertise -is the most common method for identifying properties of test items (e.g., content, format, context, language) that may cause DIF. However, expert reviews tend to be based on surface characteristics of items (e.g., Ercikan et al, 2010); whether these surface characteristics are the sources of DIF remains an empirical question. Therefore, even though expert reviews can identify whether certain aspects of test items are associated with DIF, they may not identify real sources of DIF.…”