Retrofitting Diagnostic Classification Models to Responses From IRT-Based Assessment Forms

Abstract

Developing a diagnostic tool within the diagnostic measurement framework is the optimal approach to obtain multidimensional and classification-based feedback on examinees. However, end users may seek to obtain diagnostic feedback from existing item responses to assessments that have been designed under either the classical test theory or item response theory frameworks. Retrofitting diagnostic classification models to existing assessments designed under other psychometric frameworks could be a plausible approach to obtain more actionable scores or understand more about the constructs themselves. This study (a) discusses the possibility and problems of retrofitting, (b) proposes a step-by-step retrofitting framework, and (c) explores the information one can gain from retrofitting through an empirical application example. While retrofitting may not always be an ideal approach to diagnostic measurement, this article aims to invite discussions through presenting the possibility, challenges, process, and product of retrofitting.

Observed Score Reliability Indices in Diagnostic Classification Models

“…The conditional distribution of a test score given an examinee's attribute mastery pattern reflects the mean and variance of a raw score for an examinee with that pattern. A similar argument was proposed by Liu, Huggins-Manley, and Bulut (2018) who presented the relationship between an estimated unidimensional IRT score and an attribute mastery pattern using box plots to discuss the mastery process of attributes. This approach was to one way to understand the relative difficulty of mastering each attribute.…”

Relative Diagnostic Profile: A Subscore Reporting Framework

“…Before closing this article, we want to affirm that developing a test under the diagnostic measurement framework provides higher quality RDP reports than retrofitting DCMs to current unidimensional tests. This dimensionality conundrum has been discussed in articles such as Liu et al (2017) and Sinharay et al (2011). For tests that have complex structure items (i.e., items measuring more than one attribute), fitting DCMs may be more likely to produce unstable estimates because of the high correlations among attributes.…”

“…Before employing the RDP framework, a DCM should be fitted to the item responses. We adopted the four steps in the Liu et al (2017) framework to retrofit a DCM to scored responses on a test. The four steps are (1) gathering information, (2) specifying attributes and the item-attribute relationship, (3) fitting DCMs and evaluating fit, and (4) producing and interpreting results.…”

A general diagnostic classification model for rating scales

“…DCMs can be very useful in psychological rating scales. For example, responses on a Myers-Briggs type indicator (MBTI; Myers & Myers, 1980) questionnaire could be scored using a DCM with four dichotomous attributes through which individuals are assigned with one of the 16 possible attribute profiles (e.g., ENTJ: extraversion, intuition, thinking, and judging) through a retrofitting process (Liu, Huggins-Manley, & Bulut, 2018). However, traditional DCMs for polytomous item responses require a very large sample size that is hardly attainable for either research or assessment practice in daily life.…”