Estimating Critical Values for Strength of Alignment Among Curriculum, Assessments, and Instruction

Abstract

School accountability decisions based on standardized tests hinge on the degree of alignment of the test with the state's standards documents. Yet, there exist no established criteria for judging strength of alignment. Previous measures of alignment among tests, standards, and teachers' instruction have yielded mixed results that are difficult to interpret and to compare across studies. This article reports findings from a simulation study to determine critical values for Porter's alignment index, suitable for hypothesis testing at alpha levels of .05 and .10, as is familiar to many scholars. It then reexamines the results of previous alignment studies and demonstrates how these findings may be compared through the use of these critical values. The tables of critical values will be useful to researchers or policymakers who seek to judge strength of alignment of standards, assessments, and instruction. The article also describes directions for future research in establishing objective criteria for the interpretation of alignment indices.

The challenges of alignment for the Next Generation Science Standards

“…Making an appropriate comparand by parsing the contents of source documents is part of the alignment comparison. The choice of how to parse, and why, would affect the results of a comparison; the coarser grain at which one compares, the easier it may be to find agreement between a proposed curriculum and a set of standards (Fulmer, ).…”

“…Methodologically, the biggest challenge is how to produce a meaningful interpretation for integration, whether as a quantifiable score or a qualitative judgment. Quantitatively, Fulmer et al (2011, 2014) showed how Porter's index can be interpreted statistically, but this is inadequate for the three‐dimensional nature of NGSS. Further work is needed to try out alternative alignment scoring or indices, such as attending to multivariate distance, or weighting so that quality of integration could attenuate or amplify a score that was based only on coverage.…”

“…Reading across the alignment definitions and related concepts yields two major insights. First, alignment research requires specifying a referent , the target against which a comparison is made (Fulmer, ; Porter, ). In concept, this could be the educational standards themselves, but in practice many studies adopt another framework as a basis to compare standards as well as curriculum, pedagogy, and assessment.…”

“…The choice of coding scheme that suits one set of standards may mask meaningful content distinctions in other standards (Newton & Kasten, ) or could benefit tools and approaches that are more closely associated with a selected framework (Khaliqi, ). As Fulmer () showed, the estimate of alignment is even affected substantially by the coding process decisions: simpler coding schemes are more likely to have high indices of alignment, and breaking comparands into a higher number of brief segments is more likely to give lower indices of alignment. These coding and scoring decisions thus have implications for the interpretations of degree of alignment and can change an assertion of alignment from seeming acceptable to being unacceptable (Fulmer & Polikoff, ; Polikoff & Fulmer, ).…”

Alignment between the Learning Objectives and Examinations: A Comparative Analysis of High-School Physics Curricula in Myanmar and South Australia

“…An important measure to understand the effectiveness of such content delivery and its assessment is the alignment of a curriculum with its objectives and assessment outcomes. Such studies have been carried out in United States [5][6][7] in Africa [8,9], in China [10][11][12] and in Singapore [13] and have assisted policy makers and curriculum and assessment developers to enhance their materials coherence and consistency. This paper seeks to explore such alignment in the physics curricula and examination materials in the two previously unresearched contexts of Myanmar and South Australia in comparison with previously carried out alignment studies (from New York, China and Singapore).…”