Measuring the Reliability of Diagnostic Mastery Classifications at Multiple Levels of Reporting

Abstract

As the use of diagnostic assessment systems transitions from research applications to large-scale assessments for accountability purposes, reliability methods that provide evidence at each level of reporting must are needed. The purpose of this paper is to summarize one simulation-based method for estimating and reporting reliability for an operational, large-scale, diagnostic assessment system. This assessment system reports the results and associated reliability evidence at the individual skill level for each academic content standard and broader content strands. The system also summarizes results for the overall subject using achievement levels, which are often included in state accountability metrics. Results are summarized as measures of association between true and estimated mastery status for each level of reporting.

Using Simulated Retests to Estimate the Reliability of Diagnostic Assessment Systems

“…This article expands on previous work (e.g., Roussos et al, 2007;Thompson et al, 2019) to examine how simulation-retest estimates of classification accuracy and consistency compare to other methods. Because findings were generally consistent with other methods, we argue that simulated retests may be preferred because they can estimate reliability at multiple levels of reporting, not just the skill level (Thompson et al, 2019).…”

“…This article expands on previous work (e.g., Roussos et al, 2007;Thompson et al, 2019) to examine how simulation-retest estimates of classification accuracy and consistency compare to other methods. Because findings were generally consistent with other methods, we argue that simulated retests may be preferred because they can estimate reliability at multiple levels of reporting, not just the skill level (Thompson et al, 2019). As operational programs continue to adopt DCM-based assessments, the capacity to report results and provide reliability evidence at levels beyond just the skill level is important for meeting the needs of stakeholders.…”

“…Similarly, the simulation-retest methodology can be used to estimate reliability at other levels of reporting. Thompson et al (2019) demonstrated how a simulation-retest method could be used to estimate the reliability of assessments scaled with DCMs at different levels of reporting.…”

“…Similarly, the simulation-retest methodology can be used to estimate reliability at other levels of reporting. Thompson et al (2019) demonstrated how a simulation-retest method could be used to estimate the reliability of assessments scaled with DCMs at different levels of reporting. Thompson et al applied the simulation-retest method to provide reliability evidence at multiple levels of reporting that are used for an operational, largescale state assessment.…”

“…Calculating reliability estimates. The general approach to the simulation-retest reliability method, as described by Thompson et al (2019), is to simulate a second set of responses based on actual respondent performance and calibrated-model parameters, score real-test data and simulated-test data, and compare respondents' estimated mastery statuses for the observed and simulated data. That is, once response data has been collected, calibrated, and scored, a second administration can be simulated using the known model parameters from the first (i.e., real) administration.…”