Using Differential Item Functioning Procedures to Explore Sources of Item Difficulty and Group Performance Characteristics

Abstract

Statistics used to detect differential item functioning can also reflect differential strengths and weaknesses in the performance characteristics of population subgroups. In turn, item features associated with the differential performance patterns are likely to reflect some facet of the item task and hence its difficulty, that might previously have been overlooked. In this study, several item features were identified and coded for a large number of reading comprehension items from the two admissions testing programs. Item features included subject matter content, various properties of item structure, cognitive demand indicators, and semantic content (propositional analysis). Differential item functioning was evaluated for males and females and for White and Black examinees. Results showed a number of significant relationships between item features and indicators of differential item functioning—many of which were consistent across testing programs. Implications of the results for related areas of research are discussed.

REVISING SAT®‐VERBAL ITEMS TO ELIMINATE DIFFERENTIAL ITEM FUNCTIONING

“…Technical (specialized) science material and substantive contexts drawn from science have been found to affect 10 negatively the performance of female examinees on the SAT-V (Lawrence, Curley, and McHale 1988;Lawrence and Curley 1989;Scheuneman and Gerritz 1990). A look at Table 2 reveals that C DIF for females was eliminated after the revisions were made in three of the four science items included in this study; the other item (11) showed some reduction in the MH value in Form B (-1.42), but the further revisions in Forms C and D showed a return of negative C DIF for females.…”

“…Description ofHypothesis * (and Referl!na$, ifany) Technical/specialized science terminology may negatively affect the performance of females (Lawrence, Curley, and McHale 1988;Lawrence and Curley 1989;Scheuneman and Gerritz 1990) Technical/specialized industrial arts terminology may negatively affect the performance of females (no references from research-based on empirical observation of SAT-V pretest results)…”

Examinations and the ESL student – more evidence of particular disadvantages

“…International students seem to be particularly handicapped, compared to their English peers, when assessment is based on timed, closed examinations rather than assignments (De Vita 2000) but less so on the purely multiple-choice format (Clarke, Heaney, and Gatfield 2005). It has been argued that these patterns of differential performance for ESL students might reflect an aggregation of cracks in their test-wiseness more than a clear fracture between language and assessment modality (Scheuneman and Gerritz 1990). In a similar vein Ryan (2005) cautions against assessment that tests 'the mastery of academic discourse' (99) more than mastery of learning.…”

Differential Item Functioning in Biodata: Opportunity Access as an Explanation of Gender- and Race-Related DIF

“…This literature has shown that differences across test format, even aspects as seemingly trivial as the instructions, can affect the properties of a test, including validity (Campion, Palmer, & Campion, 1997;McDaniel & Nguyen, 2001;Ployhart & Ehrhart, 2003) and reliability (Ployhart & Ehrhart, 2003). However, efforts to relate the impact of these factors to differences in the way the test measures constructs across subgroups have been largely unsuccessful (e.g., Scheuneman & Gerritz, 1990;Schmitt & Pulakos, 1998). The current study posited that one way to understand differences across subgroups at the item-level is to take a theory-driven approach.…”