Validating High‐Stakes Testing Programs

Abstract

Cronbach made the point that for validity arguments to be convincing to diverse stakeholders, they need to be based on assumptions that are credible to these stakeholders. The interpretations and uses of high‐stakes test scores rely on a number of policy assumptions about what should be taught in schools, and more specifically, about the content standards and performance standards that should be applied to students and schools. For example, a high‐school graduation test can be developed as a measure of readiness for the world of work, for college, or for citizenship and the activities of daily life. The assumptions built into the assessment need to be subjected to scrutiny and criticism if a strong case is to be made for the validity of the proposed interpretation and use.

The Adequacy of Tools for Assessing Reading Competence: A Framework and Review

“…The deliberations of the RFAC clearly reflect an emphasis on validation as both decision making (Messick, 1989) and argumentation (Kane, 2002). In fact, because so few measures met desired or optimal standards, the real value of the RFAC's work may reside in the development of procedures and criteria for evaluating the trustworthiness of reading measures.…”

“…Following the logic of Kane's (2002) interpretive argument, we referenced our framework in relation to alternative decisionmaking functions. As Kane noted, "The test is not validated, and the test scores per se are not validated.…”

“…In terms of process, the committee members all believed that the methods used to judge trustworthiness were thorough, detailed, and based on careful discussions of the evidence and technical criteria that are required to support different forms of decision making (Kane, 2002;Messick, 1989). In terms of process, the committee members all believed that the methods used to judge trustworthiness were thorough, detailed, and based on careful discussions of the evidence and technical criteria that are required to support different forms of decision making (Kane, 2002;Messick, 1989).…”

Alternate Assessments of Students with Significant Disabilities: Alternative Approaches, Common Technical Challenges

“…the unifying concept of validity that integrates content and criterion considerations into a common framework for testing rational hypotheses about theoretically relevant hypotheses" (1989, p. 1015). This view of validity still dominates, but has been expanded to embrace an evidence evaluation orientation in the most recent version of the Standards (AERA, APA, & NCME, 1999) and by psychometricians like Kane (2002). This approach to validity values evidence about the test content, response process, relations to other variables or measures, internal structure of the test, and consequences of testing.…”

“…Unfortunately, the alternate assessments currently in use in a number of states are technically flawed because of (a) poor alignment with content standards, (b) unreliable scores, (c) scores of Downloaded by [The University of Manchester Library] at 10:47 08 October 2014 sense for alternate assessment. For example, in a recent commissioned publication from the National Center on Education Outcomes (NCEO) by Gong and Marion (2006) entitled "Dealing with Flexibility in Assessments for Students With Significant Cognitive Disabilities," the authors neither cite nor used evidence frameworks from the authoritative sources on validity (e.g., the AERA, APA, & NCME Standards; Kane, 2002;Messick, 1989) while examining concerns about the lack of standardization in the administration and use of alternate assessments. Figure 1 provides an overview of how alternate assessments (both the existing alternate assessment for students with significant cognitive disabilities and the recently proposed alternate assessment for students with less severe disabilities that comprise another 2% of the total population) fit into a comprehensive statewide student assessment and accountability system.…”