Instructional Sensitivity as a Psychometric Property of Assessments

Abstract

Standards‐based reform, as codified by the No Child Left Behind Act, relies on the ability of assessments to accurately reflect the learning that takes place in U.S. classrooms. However, this property of assessments—their instructional sensitivity—is rarely, if ever, investigated by test developers, states, or researchers. In this paper, the literature on the psychometric property of instructional sensitivity is reviewed. Three categories of instructional sensitivity measures are identified—those relying on item or test scores only, those relying on item or test scores and teacher reports of instruction, and strictly judgmental methods. Each method identified in the literature is discussed alongside the evidence for its utility. Finally, recommendations are made as to the proper role of instructional sensitivity in the evaluation of assessments used under standards‐based reform.

Making Sense of Common Test Items That Do Not Get Easier Over Time: Implications for Vertical Scale Designs

“…If the items are in fact aligned to the curriculum of instruction, then p-value reversals may be an indication of test content that is not instructionally sensitive (Polikoff, 2010;Popham, 2007). In this article we demonstrate what we believe to be a relatively novel approach for investigating this issue.…”

Measurement invariance techniques to enhance measurement sensitivity

“…Such insensitivity may in turn lead to false conclusions about intervention or program effects, and result in increased research costs particularly due to the need for large samples, particularly in cluster randomised trials. Unfortunately, there is no agreement on how best to evaluate measurement sensitivity, and every current approach has limitations (Polikoff, 2010). To tackle this problem we propose the use of measurement invariance (MI) methods in a two-step process to evaluate measurement sensitivity, in accord with suggestions of such use (Millsap, 2011;Popham, 2007a;Popham and Ryan, 2012).…”

“…There is no universally agreed upon technique for estimating the responsiveness or sensitivity of an instrument to a particular intervention (Liang et al, 2002;Polikoff, 2010). However, all current practices assess responsiveness in some fashion at the total score level.…”

“…Instructional sensitivity refers to the degree to which students' performance on a test accurately reflects the quality of instruction that was provided to promote student mastery of the skill(s) being assessed. Instructionally sensitive tests should be capable of distinguishing between strong and weak instruction such that high performance on the test is attributable to good instruction (Polikoff, 2010;Popham, 2007a). Instructional sensitivity also has been labelled instructional validity, and much of the emphasis in the area of instructional sensitivity focuses on the validity of scores from instruments designed to assess instructional outcomes.…”

“…As such, they are not optimised to detect hypothesised changes in achievement that result from a specific intervention (e.g., Hevey et al, 2004) such as a new instructional technique, which may be designed to influence a small number of very specific skills, rather than the more global abilities measured by standardised instruments. Unfortunately, there is no agreement on how best to evaluate measurement sensitivity (Polikoff, 2010). Thus, the goal of this research is: a to outline an innovative solution to this problem involving new methods to evaluate measurement sensitivity b to provide a new paradigm for the development of more sensitive measures for assessing important but relatively specific outcomes that are a function of a particular intervention.…”