It's All About Context

Abstract

Assessments, even summative, are part of the learning journey not just for the student, but for the instructors as well. In competency-based education, performance assessments (PAs) are regarded as a highly authentic method of measurement, but their complexity makes them vulnerable to construct-irrelevant variance such as group bias. A differential item functioning (DIF) study was conducted to detect potential bias in a series of information technology PA tasks in which task scenarios were identified (by SMEs) as neutral or potentially controversial, where the latter was hypothesized as more likely to trigger DIF for certain demographics. Given the variety of DIF methods available and their relative strengths and weaknesses, three common statistical methods – Mantel-Haenszel (MH), logistic regression (LR), and Lord's chi-square (LC) - were used followed by a substantive review of DIF items. Hypotheses were largely supported by the analysis and review. Discussion centers on the implication of findings for assessment strategies in education.