A Review of Ets Differential Item Functioning Assessment Procedures: Flagging Rules, Minimum Sample Size Requirements, and Criterion Refinement

Zwick, Rebecca

doi:10.1002/j.2333-8504.2012.tb02290.x

Cited by 118 publications

(119 citation statements)

References 24 publications

Supporting

Mentioning

104

Contrasting

Unclassified

Order By: Relevance

“…An effect size can be constructed through a logarithmic transformation of the statistic Δα MH ¼ −2.35 lnðα MH Þ [67]. This effect size measure was adopted by the Educational Testing Service (ETS) and is called the ETS delta scale; it has been in use for over 25 years [68]. The ETS classifies jΔα MH j < 1 as negligible DIF, 1 ≤ jΔα MH j < 1.5 as small to moderate DIF, and jΔα MH j ≥ 1.5 as large DIF.…”

Section: E Dif Analysesmentioning

confidence: 99%

Gender fairness within the Force Concept Inventory

Traxler

Henderson

Stewart

et al. 2018

Phys. Rev. Phys. Educ. Res.

134

View full text Add to dashboard Cite

Research on the test structure of the Force Concept Inventory (FCI) has largely ignored gender, and research on FCI gender effects (often reported as "gender gaps") has seldom interrogated the structure of the test. These rarely crossed streams of research leave open the possibility that the FCI may not be structurally valid across genders, particularly since many reported results come from calculus-based courses where 75% or more of the students are men. We examine the FCI considering both psychometrics and gender disaggregation (while acknowledging this as a binary simplification), and find several problematic questions whose removal decreases the apparent gender gap. We analyze three samples (total N pre ¼ 5391, N post ¼ 5769) looking for gender asymmetries using classical test theory, item response theory, and differential item functioning. The combination of these methods highlights six items that appear substantially unfair to women and two items biased in favor of women. No single physical concept or prior experience unifies these questions, but they are broadly consistent with problematic items identified in previous research. Removing all significantly gender-unfair items halves the gender gap in the main sample in this study. We recommend that instructors using the FCI report the reduced-instrument score as well as the 30-item score, and that credit or other benefits to students not be assigned using the biased items.

show abstract

Section: E Dif Analysesmentioning

confidence: 99%

Gender fairness within the Force Concept Inventory

Traxler

Henderson

Stewart

et al. 2018

Phys. Rev. Phys. Educ. Res.

134

View full text Add to dashboard Cite

show abstract

“…The classification recommended by Educational Testing Service is widely recognized and employed in the field to objectively interpret DIF levels. The following are generally defined DIF levels although there could be certain changes when specific restrictions of methods are considered (Zwick, 2012 …”

Section: Graphic 1 Scree Plotmentioning

confidence: 99%

Examining Differential Item Functions of Different Item Ordered Test Forms According to Item Difficulty Levels

2016

EDUC SCI-THEOR PRACT

View full text Add to dashboard Cite

The study aims to examine whether differential item function is displayed in three different test forms that have item orders of random and sequential versions (easy-to-hard and hard-to-easy), based on Classical Test Theory (CTT) and Item Response Theory (IRT) methods and bearing item difficulty levels in mind. In the correlational research, the data from a total of 578 seventh graders were gathered using an Atomic Structures Achievement Test. R programming language and "difR" package were employed for all the analyses. As a result of the analyses, it was concluded that a comparison of IRT-and CTT-based methods indicate a greater number of items with distinctively significant differential item functioning. Different item ordering leads students at the same ability levels to display different performances on the same items. As a result, it is found that item order differentiates the probability of correct response to the items for those at the same ability levels. A test form of sequential easy-to-hard questions brings more advantages than that of a hard-to-easy sequence or a random version. The findings show that it is essential to arrange tests that are employed to make decisions about people in consideration with psychometric principles.

show abstract

“…But it could be done better. Zwick (2012) reviewed the status of ETS DIF analysis procedures, focusing on three aspects: (a) the nature and stringency of the statistical rules used to flag items, (b) the minimum sample size requirements that are currently in place for DIF analysis, and (c) the efficacy of criterion refinement. Recommendations were made with respect to improved flagging rules, minimum sample size requirements, and procedures for combining data across administrations.…”

Section: Limitations Of Quantitative Fairness Assessment Proceduresmentioning

confidence: 99%

Contributions to the Quantitative Assessment of Item, Test, and Score Fairness

Dorans¹

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

A Review of Ets Differential Item Functioning Assessment Procedures: Flagging Rules, Minimum Sample Size Requirements, and Criterion Refinement

Cited by 118 publications

References 24 publications

Gender fairness within the Force Concept Inventory

Gender fairness within the Force Concept Inventory

Examining Differential Item Functions of Different Item Ordered Test Forms According to Item Difficulty Levels

Contributions to the Quantitative Assessment of Item, Test, and Score Fairness

Contact Info

Product

Resources

About