A Framework for Developing Cognitively Diagnostic Assessments

Nichols, Paul D.

doi:10.2307/1170588

Cited by 16 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, when heterogeneous pools of items are scaled and a student is said to have a score of 190 on "data and probability," there is not an obvious match to a teaching model that can be defended psychometrically (even if Rasch assumptions are satisfied). Lack of concrete guidance was especially disappointing given important earlier work by Nichols (1994) in which he distinguished between traditional measurement designs, which average over a test taker's responses, and cognitively diagnostic assessments, which are based on a carefully developed substantive model of the domain. Nichols et al intend to answer my questions about specific instantiation of their model by talking through an algebra tutoring example, but this too was disappointing.…”

Section: Formative Assessment Validity Framework: Nichols Meyers Anmentioning

confidence: 99%

Commentary: Evaluating the Validity of Formative and Interim Assessment

Shepard

2009

Educational Measurement

View full text Add to dashboard Cite

In many school districts, the pressure to raise test scores has created overnight celebrity status for formative assessment. Its powers to raise student achievement have been touted, however, without attending to the research on which these claims were based. Sociocultural learning theory provides theoretical grounding for understanding how formative assessment works to increase student learning. The articles in this special issue bring us back to underlying first principles by offering separate validity frameworks for evaluating formative assessment (Nichols, Meyers, & Burling) and newly‐invented interim assessments (Perie, Marion, & Gong). The article by Heritage, Kim, Vendlinski, and Herman then offers the most important insight of all; that is, formative assessment is of little use if teachers don't know what to do when students are unable to grasp an important concept. While it is true that validity investigations are needed, I argue that the validity research that will tell us the most—about how formative assessment can be used to improve student learning—must be embedded in rich curriculum and must at the same time attempt to foster instructional practices consistent with learning research.

show abstract

Section: Formative Assessment Validity Framework: Nichols Meyers Anmentioning

confidence: 99%

Commentary: Evaluating the Validity of Formative and Interim Assessment

Shepard

2009

Educational Measurement

View full text Add to dashboard Cite

show abstract

“…Many researchers and practitioners in the psychometric community agree that the psychology of test performance must be understood to develop, score, and validly interpret results from educational tests (e.g., Frederiksen, Mislevy, & Bejar, 1993;Gierl, 1997;Hattie, Jaeger, & Bond, 1999;Mislevy, 1996;Nichols, 1994;Nichols, Chipman, & Brennan, 1995;Snow & Lohman, 1989;Wiley, 1991). Consensus on this view seems inevitable because most educational tests are based on cognitive problem-solving tasks.…”

Section: Future Directionsmentioning

confidence: 99%

Identifying Sources of Differential Item and Bundle Functioning on Translated Achievement Tests: A Confirmatory Analysis

Gierl

Khaliq

2001

J Educational Measurement

View full text Add to dashboard Cite

Increasingly, tests are being translated and adapted into different languages. Differential item functioning (DIF) analyses are often used to identify non‐equivalent items across language groups. However, few studies have focused on understanding why some translated items produce DIF. The purpose of the current study is to identify sources of differential item and bundle functioning on translated achievement tests using substantive and statistical analyses. A substantive analysis of existing DIF items was conducted by an 11‐member committee of testing specialists. In their review, four sources of translation DIF were identified. Two certified translators used these four sources to categorize a new set of DIF items from Grade 6 and 9 Mathematics and Social Studies Achievement Tests. Each item was associated with a specific source of translation DIF and each item was anticipated to favor a specific group of examinees. Then, a statistical analysis was conducted on the items in each category using SIBTEST. The translators sorted the mathematics DIF items into three sources, and they correctly predicted the group that would be favored for seven of the eight items or bundles of items across two grade levels. The translators sorted the social studies DIF items into four sources, and they correctly predicted the group that would be favored for eight of the 13 items or bundles of items across two grade levels. The majority of items in mathematics and social studies were associated with differences in the words, expressions, or sentence structure of items that are not inherent to the language and/or culture. By combining substantive and statistical DIF analyses, researchers can study the sources of DIF and create a body of confirmed DIF hypotheses that may be used to develop guidelines and test construction principles for reducing DIF on translated tests.

show abstract

“…Over the past two decades, measurement specialists have shown growing interest in the development of CDMs with an attempt to incorporate cognitive models into test design and analysis so test results provide more diagnostic information about student thinking and learning. Designing test items according to a cognitive model has been recognized as an important way to improve the quality of test items and the validity of inferences drawn from tests (Borsboom, 2005;Embretson & Gorin, 2001;Leighton, 2004;Mislevy, 2006;Nichols, 1994). This is because a cognitive model provides an explicit understanding of the knowledge and skills normally used by students to solve standardized tasks in a test domain.…”

mentioning

confidence: 99%

The Hierarchy Consistency Index: Evaluating Person Fit for Cognitive Diagnostic Assessment

Cui

Leighton

2009

J Educational Measurement

View full text Add to dashboard Cite

In this article, we introduce a person‐fit statistic called the hierarchy consistency index (HCI) to help detect misfitting item response vectors for tests developed and analyzed based on a cognitive model. The HCI ranges from −1.0 to 1.0, with values close to −1.0 indicating that students respond unexpectedly or differently from the responses expected under a given cognitive model. A simulation study was conducted to evaluate the power of the HCI in detecting different types of misfitting item response vectors. Simulation results revealed that the detection rate of the HCI was a function of type of misfit, item discriminating power, and test length. The best detection rates were achieved when the HCI was applied to tests that consisted of a large number of highly discriminating items. In addition, whether a misfitting item response vector can be correctly identified depends, to a large degree, on the number of misfits of the item response vector relative to the cognitive model. When misfitting response behavior only affects a small number of item responses, the resulting item response vector will not be substantially different from the expectations under the cognitive model and consequently may not be statistically identified as misfitting. As an item response vector deviates further from the model expectations, misfits are more easily identified and consequently higher detection rates of the HCI are expected.

show abstract

A Framework for Developing Cognitively Diagnostic Assessments

Cited by 16 publications

References 0 publications

Commentary: Evaluating the Validity of Formative and Interim Assessment

Commentary: Evaluating the Validity of Formative and Interim Assessment

Identifying Sources of Differential Item and Bundle Functioning on Translated Achievement Tests: A Confirmatory Analysis

The Hierarchy Consistency Index: Evaluating Person Fit for Cognitive Diagnostic Assessment

Contact Info

Product

Resources

About