A Validity Argument in Support of the Use of College Admissions Test Scores for Federal Accountability

Abstract

In 2018, 26 states administered a college admissions test to all public school juniors. Nearly half of those states proposed to use those scores as their academic achievement indicators for federal accountability under the Every Student Succeeds Act (ESSA); many others are planning to use those scores for other accountability purposes. Accountability encompasses a number of different uses and subsumes a variety of claims. For states proposing to use summative tests for accountability, a validity argument needs to be developed, which entails delineating each specific use of test scores associated with accountability, identifying appropriate evidence, and offering a rebuttal to counterclaims. The aim of this article is to support states in developing a validity argument for use of college admission test scores for accountability by identifying claims that are applicable across states, along with summarizing existing evidence as it relates to each of these claims. As outlined by The Standards for Educational and Psychological Testing, multiple sources of evidence are used to address each claim. A series of threats to the validity argument, including weaker alignment with content standards and potential influences in narrowing teaching, are reviewed. Finally, the article contrasts validity evidence, primarily from research on the ACT, with regulatory requirements from ESSA. The Standards and guidance addressing the use of a “nationally recognized high school academic assessment” (Elementary and Secondary Education Act (ESEA), Negotiated Rulemaking Committee; Department of Education) are the primary sources for the organization of validity evidence.

An Argument in Search of Evidence: A Response to Camara et al. (2019)

“…Second, even if one evaluates the Camara et al. () argument against a more limited set of claims, we find the evidence incomplete and/or overstated. We offer several additional comments regarding the construction of their validity argument, the lack of transparency for evaluating many of the claims, and the potential unintended negative consequences associated with the use of the ACT or SAT as the primary achievement indicator in a state accountability system.…”

“…Camara et al.’s () first claim in their validity argument focuses on the college readiness tests being “relevant to success in high school and college.” They put forth arguments related to four of the five sources of validity evidence articulated in the joint standards (AERA, APA, & NCME, ): content, internal structure, and response processes in support of this claim. They reserve their limited discussion of consequences to their fourth claim.…”

“…Camara et al. () put forth some evidence that the ACT is aligned (or aligned enough at least) to most state content standards. We have serious concerns with the nature and quality of this evidence.…”

“…We appreciate Camara, Mattern, Croft, and Vispoel () in their attempt to outline a validity argument to evaluate the use of college‐admission tests in high school accountability systems under the Every Student Succeeds Act (ESSA). We believe, however, their argument and evidence fall short in several important ways.…”

“…We focus the remainder of this commentary on the evidence Camara et al. () offered in support of college admission tests as indicators of high school achievement.…”