How Fair Is My Test?

Abstract

Abstract. The degree to which test scores can support justified and fair decisions about demographically diverse participants has been an important aspect of educational and psychological testing for millennia. In the last 30 years, this aspect of measurement has come to be known as consequential validity, and it has sparked scholarly debate as to how responsible psychometricians should be for the fairness of the tests they create and how the field might be able to quantify that fairness and communicate it to applied researchers and other stakeholders of testing programs. Here, we formulate a relatively simple-to-calculate ratio coefficient that is meant to capture how well the scores from a given test can predict a criterion free from the undue influence of student demographics. We posit three example calculations of this Consequential Validity Ratio (CVR): one where the CVR is quite strong, another where the CVR is more moderate, and a third where the CVR is weak. We provide preliminary suggestions for interpreting the CVR and discuss its utility in instances where new tests are being developed, tests are being adapted to a new population, or the fairness of an established test has become an empirical question.

Evaluating the Consequential Validity of the Research-Based Early Mathematics Assessment

“…In contrast, the scale-level evidence was produced via the Consequential Validity Ratio (CVR, Dumas et al, 2022). Although recently formalized, this index was inspired by past regressionbased methods for examining test fairness (e.g., Millsap, 2011).…”

“…The current study applies a new and efficient method (i.e., CVRs) to produce scale-level evidence for the justification of consequential validity. Notably, although CVRs are straightforward to calculate and relatively easy to interpret, they are not recommended to be used to replace or skip over other necessary validation efforts (Dumas et al, 2022), such as the DIF analyses shown in this study.…”

“…Consequential validity, or the degree of score uses and interpretations that can support justified and fair decisions (Messick, 1995), has been recognized as an essential aspect of testing practices by many researchers and psychological professionals (e.g., AERA/APA/NCME, 2014; Dumas et al, 2022; Iliescu & Greiff, 2021). The REMA is a major measurement tool in early childhood research, and previous studies have demonstrated its internal and external validity using a variety of methods (Clements et al, 2008b; Dong et al, 2021).…”

What Makes Children's Responses to Creativity Assessments Difficult to Judge Reliably?

“…The same could potentially be said of younger students, or students with special education or gifted identification, which could lead the judges to disagree with one another resulting in larger rater variance in scoring. In the United States, race groups are a particularly sensitive grouping variable across which educational and psychological researchers work hard to make valid and fair inferences (see Dumas, Dong, & McNeish, 2022 for one psychometric perspective). Because race groupings could indicate cultural differences that may affect how students express their ideas and formulate their responses, therefore, potentially creating larger rater variance for certain race/ethnicity groups than others.…”

Promoting inclusive recruiting and selection into military training schools: Admission waivers versus retesting.

“…Consequential validity differs from the validity definition within the diversity-validity dilemma because it conceptualizes societally beneficial usage of test scores as a direct component of validity rather than a potential tradeoff of external predictive ability. Although consequential validity research does not disregard relations among test scores and external criteria, it does suggest that such relations may be moderated by demographic variables and that moderation effects are relevant for validity arguments (e.g., Dumas et al, 2022). In the current joint standards for educational and psychological testing, both relations with external criteria and societal effects of testing are considered acceptable sources of validity evidence (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014, ch.…”