Exploring the Impact of Q-Matrix Specifications Through a DINA Model in a Large-Scale Mathematics Assessment

Abstract

The demand for diagnostic feedback has triggered extensive research on cognitive diagnostic models (CDMs), such as the deterministic input, noisy output “and” gate (DINA) model. This study explored two Q-matrix specifications with the DINA model in a statewide large-scale mathematics assessment. The first Q-matrix was developed based on five predefined content reporting categories, and the second was based on the post hoc coding of 15 attributes by test-development experts. Total raw scores correlated strongly with the number of skills mastered, using both Q-matrices. Correlations between the DINA-model item statistics and those from the item response theory analyses were moderate to strong, but were always lower for the 15-skill model. Results highlighted the trade-off between finer-grained modeling and less precise model estimation.

The Development of an Instrument for Longitudinal Learning Diagnosis of Rational Number Operations Based on Parallel Tests

“…Currently, there are many applications use cross-sectional LDMs to diagnose individuals’ learning status in the field of mathematics because the structure of mathematical attributes is relative clear to be identified, such as fraction calculations ( Tatsuoka, 1983 ; Wu, 2019 ), linear algebraic equations ( Birenbaum et al, 1993 ), and spatial rotations ( Chen et al, 2018 ; Wang et al, 2018 ). Some studies also apply cross-sectional LDMs to analyze data from large-scale mathematical assessments (e.g., George and Robitzsch, 2018 ; Park et al, 2018 ; Zhan et al, 2018 ; Wu et al, 2020 ). However, most of these application studies use cross-sectional design and cannot track the individual growth of mathematical ability.…”

Cognitive diagnosis and algorithmic cultural analysis of fourth-grade Yi students’ mathematical skills in China: A case study of several primary schools in Puge county, Liangshan prefecture

“…Latent classification models are designed to classify the subject and help the subject find its place in the group. This test uses the DINA model ( Henson, 2005 ; Wu et al, 2020 ), which is a latent classification model that allows the data collected from student responses to be processed to analyze the individual’s knowledge status and mastery of cognitive attributes behind the scores ( de la Torre, 2011 ). Compared to other potential classification models, the DINA model not only portrays the mental processes of students during arithmetic, but also involves only two parameters, the miss parameter and the guess parameter, which are easy to implement for estimation of these two parameters.…”

Psychological Studies in the Teaching, Learning and Assessment of Mathematics

“…One of the outstanding characteristics of CDA is its assessment structure. Through a more operational and internally consistent Q-matrix, subjects' unobservable cognitive state can be linked to the observable item responses, which goes beyond the simple two-way list format Frontiers in Psychology 05 frontiersin.org (Wu et al, 2020). In this study, based on the final Q-matrix in Table 3, we selected 60 items from one of the large-scaled assessment tests Trends in International Mathematics and Science Study (TIMSS) as the first round of items selected.…”