Developing and evaluating instructionally sensitive assessments in science

Abstract

The purpose of this article is to address a major gap in the instructional sensitivity literature on how to develop instructionally sensitive assessments. We propose an approach to developing and evaluating instructionally sensitive assessments in science and test this approach with one elementary lifescience module. The assessment we developed was administered to 125 students in seven classrooms. The development approach considered three dimensions of instructional sensitivity; that is, assessment items should: represent the curriculum content, reflect the quality of instruction, and have formative value for teaching. Focusing solely on the first dimension, representation of the curriculum content, this study was guided by the following research questions: (1) What science module characteristics can be systematically manipulated to develop items that prove to be instructionally sensitive? and (2) Are the instructionally sensitive assessments developed sufficiently valid to make inferences about the impact of instruction on students' performance? In this article, we describe our item development approach and provide empirical evidence to support validity arguments about the developed instructionally sensitive items. Results indicated that: (1) manipulations of the items at different proximities to vary their sensitivity were aligned with the rules for item development and also corresponded with pre-to-post gains; and (2) the items developed at different distances from the science module showed a pattern of pre-to-post gain consistent with their instructional sensitivity, that is, the closer the items were to the science module, the larger the observed gains and effect sizes. ß 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 2012

Case‐based instruction: Improving students' conceptual understanding through cases in a mechanical engineering course

“…Given that learning measures need to be sensitive to gauge differences between case‐based and traditional lecture, it is important that assessment “fit the philosophy of active learning rather than passive reproductive learning” (Reynolds, , p. 22). Ruiz‐Primo et al () also argued for the need to develop instructionally sensitive assessments that reflect not only the content being taught, but also the quality of instruction. Previous research has suggested that open‐ended assessments that measure conceptual understanding are more likely to show influence of student‐centered instruction than fact‐based assessments (Gallagher, ).…”

Modeling Instructional Sensitivity Using a Longitudinal Multilevel Differential Item Functioning Approach

“…Haladyna and Roid () as well as Polikoff () emphasize the use of PPDI because it is technically easy to implement and conceptually easy to understand. In an experimental study, Ruiz‐Primo and colleagues () found items’ PPDI to be proportional to the alignment of item characteristics and the implemented curriculum. Also, item selection based on PPDI does not negatively impact reliability (Crehan, ).…”

Conducting causal effects studies in science education: Considering methodological trade‐offs in the context of policies affecting research in schools

“…Oftentimes, it is the researchers themselves who would know best how to make instruments sufficiently sensitive to the treatment effect. One approach to reconciling this conflict is to employ a series of assessments or subscales that range in distance from the students' experiences with the intervention (Ruiz‐Primo, Shavelson, Hamilton, & Klein, , Ruiz‐Primo et al, ). That is, the assessments include a set of “close” items that are very closely aligned with students' experiences with the intervention; “proximal” items that are aligned with the learning objectives of the intervention; and distal items that are aligned with national or state standards.…”