Instructional Sensitivity of a State's Standards-Based Assessment

Abstract

The accuracy of achievement test score inferences largely depends on the sensitivity of scores to instruction focused on tested objectives. Sensitivity requirements are particularly challenging for standards-based assessments because a variety of plausible instructional differences across classrooms must be detected. For this study, we developed a new method for capturing the alignment between how teachers bring standards to life in their classrooms and how the standards are defined on a test. Teachers were asked to report the degree to which they emphasized the state's academic standards, and to describe how they taught certain objectives from the standards. Two curriculum experts judged the alignment between how teachers brought the objectives to life in their classrooms and how the objectives were operationalized on the state test. Emphasis alone did not account for achievement differences among classrooms. The best predictors of classroom achievement were the match between how the standards were taught and tested, and the interaction between emphasis and match, indicating that test scores were sensitive to instruction of the standards, but in a narrow sense.

Alignment between the physics content standard and the standardized test: A comparison among the United States‐New York State, Singapore, and China‐Jiangsu

“…Research has shown that simply covering topics is not a good predictor for student achievement on standardized tests; it is the coverage and the cognitive emphases together that predict significantly student performances on state standardized tests (D'Agostino, Welsh, & Corson, 2007;Gamoran, Porter, Smithson, & White, 1997). In other words, when a teacher interprets and implements the intended content standard in terms of the content and cognitive emphases in the same way as the standardized test, students are more likely to perform well on the standardized test.…”

Combining Multiple Measures of Students' Opportunities to Develop Analytic, Text-Based Writing Skills

“…Our hypotheses stem from the literature demonstrating significant findings for instructional covariates where there is high overlap between what is measured in instruction and the student assessment (D'agostino, Welsh, & Corson, 2007;Leinhardt & Seewald, 1981;Ruiz-Primo, Shavelson, Hamilton, & Klein, 2002). Because students' opportunities to develop analytical text-based writing skills had significant overlap with the RTA, we were especially interested in the relationship between OTL and student performance on the RTA.…”

Instructional Sensitivity as a Psychometric Property of Assessments

“…The term “instructional sensitivity” was chosen as the focal term for this analysis to refer to the extent to which student performance on a test or item reflects the instruction received (Kosecoff & Klein, 1974). Instead, the term “instructional validity” could have been chosen, as some authors use the terms interchangeably (e.g., D’Agostino, Welsh, & Corson, 2007). While instructional sensitivity is sometimes used to refer to the ability to detect differences in the quality of instruction (Popham, 2007), most uses of the term focus on instruction generally—implicitly, both content and quality (D’Agostino et al, 2007; Haladyna & Roid, 1981; Kosecoff & Klein, 1974; Muthen, Kao, & Burstein, 1991).…”

“…However, in both cases, instructional validity was focused on whether instruction was adequate to justify inferences and consequences made based on the results of a test. As D’Agostino et al (2007) point out, the meaning of instructional validity quickly switched to a definition essentially identical to sensitivity: “the extent to which an assessment is systematically sensitive to the nature of instruction offered” (Yoon & Resnick, 1998, p. 2), allowing researchers the ability to use the terms interchangeably. Instructional validity was the more commonly used term throughout the 1980s and 1990s.…”