Modeling Partial Knowledge on Multiple‐Choice Items Using Elimination Testing

Abstract

Single-best answers to multiple-choice items are commonly dichotomized into correct and incorrect responses, and modeled using either a dichotomous item response theory (IRT) model or a polytomous one if differences among all response options are to be retained. The current study presents an alternative IRT-based modeling approach to multiple-choice items administered with the procedure of elimination testing, which asks test-takers to eliminate all the response options they consider to be incorrect. The partial credit model is derived for the obtained responses. By extracting more information pertaining to test-takers' partial knowledge on the items, the proposed approach has the advantage of providing more accurate estimation of the latent ability. In addition, it may shed some light on the possible answering processes of test-takers on the items. As an illustration, the proposed approach is applied to a classroom examination of an undergraduate course in engineering science.

Modeling Partial Knowledge in Multiple-Choice Cognitive Diagnostic Assessment

“…was buying 2. was looking 3. is eating 4. cooks the ability to eliminate some, but not all, incorrect choices, thus restricting guessing to a proper subset of options that include the correct option (Frary, 1980;Frary & Hutchinson, 1982). Although previous studies have developed other procedures to evaluate partial knowledge, such as elimination testing (Coombs et al, 1956;Wu et al, 2019) and probability testing (Barr & Burke, 2013;Hassmén & Hunt, 1994), we focus on multiple-choice assessments and consider them within the CDM framework. It should be noted that the concept of partial knowledge differs from that of partial mastery (Shang et al, 2021;Zhan et al, 2020), which considers ordinal mastery levels within each attribute.…”

“…The literature has revealed that examinees frequently exclude some options from the possible answers based on their partial knowledge before guessing (e.g., Lindquist & Hoover, 2015). This behavior is referred to as educated guessing (Wu et al, 2021) or informed guessing (Tsai & Lin, 2017;Wu et al, 2019). Fitting the MC-DINA or MC-S-DINA models to the responses associated with partial knowledge would lead to the overestimation of attribute mastery states because the models that do not consider partial knowledge tend to under-evaluate the probability of correctly answering the items.…”