Validity theory: Reform policies, accountability testing, and consequences

Abstract

Educational policies such as Race to the Top in the USA affirm a central role for testing systems in government-driven reform efforts. Such reform policies are often referred to as the global education reform movement (GERM). Changes observed with the GERM style of testing demand socially engaged validity theories that include consequential research. The article revisits the Standards and Kane’s interpretive argument (IA) and argues that the role envisioned for consequences remains impoverished. Guided by theory of action, the article presents a validity framework, which targets policy-driven assessments and incorporates a social role for consequences. The framework proposes a coherent system that makes explicit the interconnections among policy ambitions, testing functions, and the levels/sectors that are affected. The article calls for integrating consequences into technical quality documentation, demands a more realistic delineation of stakeholders and their roles, and compels engagement in policy research.

Real World Compromises: Policy and Practice Impacts of Kindergarten Entry Assessment‐Related Validity and Reliability Challenges

“…As mentioned earlier, if a theory of action undergirds the use a test, researchers have argued that investigations of the test's validity and reliability for specific purposes should inform relevant educational policies (Bennett, ; Bennett et al, ; Chalhoub‐Deville, ). To better understand this study's implications for education policy makers, it is helpful to first revisit the sample of seven KEAs used.…”

“…They also will vary based on the measurement approach (e.g., direct assessment or observational rubric), the different individuals and systems involved in the entire testing and data use process, how often an assessment is administered, and the purposes for which the measure is being used (Mislevy, Wilson, Ercikan, & Chudowsky, ; Snow & Van Hemel, ). Furthermore, recent scholarship has argued that if a theory of action undergirds the use a test (e.g., KEA data will inform teachers' instruction and contribute to policy makers' efforts to close achievement gaps), such research should not only investigate whether that utility hypothesis is being realized but also inform relevant assessment policies (Bennett, ; Bennett et al, ; Chalhoub‐Deville, ).…”

“…However, because the administration of KEAs is now under way in many states, there is the opportunity to take a retrospective look at the on‐the‐ground, validity‐ and reliability‐relevant issues that contributed to policy and practice adjustments. By exploring these factors across different KEA contexts, there is the potential to better understand the shaping role they can play and to highlight the value of validity‐ and reliability‐related research for informing assessment policies and practices (Chalhoub‐Deville, ). Such information could be especially useful as states continue to use KEAs to inform a variety of policy‐making purposes (ELCTA, ), but presumably without the financial support of RTT‐ELC and other federal funds (Weisenfeld, ).…”

An Analysis of High School Mission Statements in Massachusetts from 2001 to 2019

“…In the past decade a seemingly endless string of educational reform efforts have been levied at the national, state, and local levels to address a myriad of perceived problems, inadequacies, and short-comings of the status quo educational system. Determining the effectiveness and impacts of such educational reform efforts and their potential influence on actual teaching and learning practices has been the concern of much modern educational research and policy literature (e.g., Chalhoub-Deville, 2016;Croft, Roberts, & Stenhouse, 2015). For a number of reasons, the success of most educational reform efforts have been studied using student performance on standardsbased assessments as the outcome measure.…”

The Government of Canada’s Second Language Evaluation, Test of Written Expression: Exploring Validity

“…Furthermore, Messick emphasizes that it is not enough to have a good instrument; it needs to be used appropriately. Since Messick's work, many researchers have studied consequences of testing both on the smaller scale of washback (see for example, Alderson & Hamp-Lyons, 1996;Hamp-Lyons;1997; and the larger scale of impact (Chalhoub-Deville, 2016;McNamara 2007;McNamara & Roever, 2006;McNamara & Ryan, 2011;Shohamy, 2007;Moss, 1998;, the topic of which is elaborated in the Washback and Impact section (see section 2.2).…”