The Hierarchy Consistency Index: Evaluating Person Fit for Cognitive Diagnostic Assessment

Abstract

In this article, we introduce a person‐fit statistic called the hierarchy consistency index (HCI) to help detect misfitting item response vectors for tests developed and analyzed based on a cognitive model. The HCI ranges from −1.0 to 1.0, with values close to −1.0 indicating that students respond unexpectedly or differently from the responses expected under a given cognitive model. A simulation study was conducted to evaluate the power of the HCI in detecting different types of misfitting item response vectors. Simulation results revealed that the detection rate of the HCI was a function of type of misfit, item discriminating power, and test length. The best detection rates were achieved when the HCI was applied to tests that consisted of a large number of highly discriminating items. In addition, whether a misfitting item response vector can be correctly identified depends, to a large degree, on the number of misfits of the item response vector relative to the cognitive model. When misfitting response behavior only affects a small number of item responses, the resulting item response vector will not be substantially different from the expectations under the cognitive model and consequently may not be statistically identified as misfitting. As an item response vector deviates further from the model expectations, misfits are more easily identified and consequently higher detection rates of the HCI are expected.

The role of formative feedback in promoting higher order thinking skills in classrooms: a theoretical model

“…Several methods for addressing this challenge have been proposed in the literature. Some of these methods are: (a) statistical analysis using the hierarchy consistency index (Cui & Leighton, 2009); (b) empirical procedures that yield verbal report data reflecting misconceptions in students' thinking and problem solving, and analysis of the errors patterns detected ; and (c) think-aloud procedures (Leighton, 2011). Though some of these procedures may appear technical and challenging to some teachers and researchers, they still hold a lot of promise if students thinking processes and learning errors are to be understood and identified.…”

Use of Full Hierarchy Consistency Index to Assess Response Consistency

“…Probability of an examinee's correct response to a test item is modeled as a function of item parameters and examinee's mastery of the attributes (Cui & Leighton, 2009). For example, the DINA model assumes that an examinee correctly responds to an item as long as the examinee has mastered all the required attributes required for that item.…”

“…Measurement accuracy of examinees is directly related to appropriateness of measurement model, which need to properly delineate the real aspect of examinees' response processes (Cui & Leighton, 2009). For instance, when attributes hold a hierarchical structure (i.e., some of the attributes are prerequisite to master others), not all 2 latent classes are permissible.…”

“…Furthermore, to avoid invalid inferences, fit of examinees' response data to the model is studied through 'person-fit' statistics. By means of person-fit statistics, examinees who are not being measured well by the test are identified (Cui & Leighton, 2009). Misfit between the examinee response data and measurement model may be due to invalid models and/or examinee's aberrant response behavior (e.g., cheating, creative responding, and random responding).…”

“…Misfit between the examinee response data and measurement model may be due to invalid models and/or examinee's aberrant response behavior (e.g., cheating, creative responding, and random responding). Cui and Leighton (2009) have introduced a person-fit index to assess classification reliability of specific cognitive diagnosis models (e.g., attribute hierarchy model [AHM: Leighton, Gierl, & Hunka, 2004]). This person-fit index is referred to as hierarchy consistency index (HCI) as it was also used by Cui (2007) to measure the accuracy of specified hierarchical structure of attributes in AHM.…”