Estimating Item Difficulty With Comparative Judgments

Abstract

Previous investigations of the ability of content experts and test developers to estimate item difficulty have, for the most part, produced disappointing results. These investigations were based on a noncomparative method of independently rating the difficulty of items. In this article, we argue that, by eliciting comparative judgments of difficulty, judges can more accurately estimate item difficulties. In this study, judges from different backgrounds rank ordered the difficulty of SAT® mathematics items in sets of 7 items. Results showed that judges are reasonably successful in rank ordering several items in terms of difficulty, with little variability across judges and content areas. Simulations of a possible implementation of comparative judgments for difficulty estimation show that it is possible to achieve high correlations between true and estimated difficulties with relatively few comparisons. Implications of these results for the test development process are discussed.

Development and Validation of a Chinese Parental Health Literacy Questionnaire for Caregivers of Children 0 to 3 Years Old

“…Item discrimination is examined using the question-total correlation [18]. A question should be deleted, when: a) item difficulty lower than 0.2 or higher than 0.8 [19,20]; and b) the coefficient of questiontotal correlation lower than 0.3 [18].…”

Development and validation of a Chinese parental health literacy questionnaire for caregivers of children 0 to 3 years old

“…Item discrimination is examined using the question-total correlation [19]. A question should be deleted, when: a) item difficulty lower than 0.2 or higher than 0.8 [20, 21]; and b) the coefficient of question-total correlation lower than 0.3 [19].…”

Development and Validation of a Chinese Parental Health Literacy Questionnaire for Caregivers of Children from 0 to 3 Years Old

“…Item discrimination is examined using the question-total correlation [19]. A question is considered to be deleted, when: a) item difficulty lower than 0.2 or higher than 0.8 [20,21]; and b) the coefficient of question-total correlation lower than 0.3 [19]. The results were shown in Additional file 1.…”