Bayesian Psychometrics for Diagnostic Assessments: A Proof of Concept

Thompson, William J.

doi:10.35542/osf.io/jzqs8

Cited by 7 publications

(5 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, many best practices for educational assessments (e.g., AERA et al, 2014) have an implicit assumption that assessments result in a scale score. Therefore, many of the standards recommend evidence that either is inappropriate or requires adaptations for diagnostic assessments (Thompson et al, 2021). Perhaps unsurprisingly, this implicit assumption has also crept into federal regulations for accountability assessments, for example, in comparability requirements for states attempting to implement innovative assessments (Marion, 2023;Rupp, 2023).…”

Section: Discussionmentioning

confidence: 99%

Improving Instructional Decision-Making Using Diagnostic Classification Models

Thompson,

Clark

2024

Preprint

View full text Add to dashboard Cite

For assessments to fulfill their purpose, the psychometric model must align with the intended uses of the assessment. In recent years, educators, administrators, and policymakers have called for assessments that require less time, but also provide more actionable results to aid stakeholders in making decisions. One promising approach to measurement to support instructional decision-making is diagnostic classification models (DCMs). DCMs are flexible psychometric models that facilitate fine-grained reporting on skills that students have mastered. In this paper, we provide a high-level overview of DCMs. We then describe different methods for reporting results from DCM-based assessments that support decision-making for different stakeholder groups. We close with a discussion of considerations for implementing DCMs in an operational setting and future directions for research.

show abstract

Section: Discussionmentioning

confidence: 99%

Improving Instructional Decision-Making Using Diagnostic Classification Models

Thompson,

Clark

2024

Preprint

View full text Add to dashboard Cite

show abstract

Section: Discussionmentioning

confidence: 99%

Improving Instructional Decision‐Making Using Diagnostic Classification Models

Thompson,

Clark

2024

Educational Measurement

View full text Add to dashboard Cite

In recent years, educators, administrators, policymakers, and measurement experts have called for assessments that support educators in making better instructional decisions. One promising approach to measurement to support instructional decision‐making is diagnostic classification models (DCMs). DCMs are flexible psychometric models that facilitate fine‐grained reporting on skills that students have mastered. In this article, we describe how DCMs can be leveraged to support better decision‐making. We first provide a high‐level overview of DCMs. We then describe different methods for reporting results from DCM‐based assessments that support decision‐making for different stakeholder groups. We close with a discussion of considerations for implementing DCMs in an operational setting, including how they can inform decision‐making at state and local levels, and share future directions for research.

show abstract

“…Absolute fit was assessed through the χ 2 obs statistic, calculated from the posterior predictive model checks (see Thompson, 2019). For this statistic, a ppp value of less than 0.05 generally indicates insufficient fit of the model to the observed data.…”

Section: Methods 1: Patterns Of Mastery Profilesmentioning

confidence: 99%

A Diagnostic Framework for the Empirical Evaluation of Learning Maps

Thompson

Nash

2022

Front. Educ.

Self Cite

View full text Add to dashboard Cite

Learning progressions and learning map structures are increasingly being used as the basis for the design of large-scale assessments. Of critical importance to these designs is the validity of the map structure used to build the assessments. Most commonly, evidence for the validity of a map structure comes from procedural evidence gathered during the learning map creation process (e.g., research literature, external reviews). However, it is also important to provide support for the validity of the map structure with empirical evidence by using data gathered from the assessment. In this paper, we propose a framework for the empirical validation of learning maps and progressions using diagnostic classification models. Three methods are proposed within this framework that provide different levels of model assumptions and types of inferences. The framework is then applied to the Dynamic Learning Maps® alternate assessment system to illustrate the utility and limitations of each method. Results show that each of the proposed methods have some limitations, but they are able to provide complementary information for the evaluation of the proposed structure of content standards (Essential Elements) in the Dynamic Learning Maps assessment.

show abstract

Bayesian Psychometrics for Diagnostic Assessments: A Proof of Concept

Cited by 7 publications

References 37 publications

Improving Instructional Decision-Making Using Diagnostic Classification Models

Improving Instructional Decision-Making Using Diagnostic Classification Models

Improving Instructional Decision‐Making Using Diagnostic Classification Models

A Diagnostic Framework for the Empirical Evaluation of Learning Maps

Contact Info

Product

Resources

About