Ongoing issues in test fairness

Abstract

In the attempt to identify or prevent unfair tests, both quantitative analyses and logical evaluation are often used. For the most part, fairness evaluation is a pragmatic attempt at determining whether procedural or substantive due process has been accorded to either a group of test takers or an individual. In both the individual and comparative approaches to test fairness, counterfactual reasoning is useful to clarify a potential charge of unfairness: Is it plausible to believe that with an alternative assessment (test or item) or under different test conditions an individual or groups of individuals may have fared better? Beyond comparative questions, fairness can also be framed by moral and ethical choices. A number of ongoing issues are evaluated with respect to these topics including accommodations, differential item functioning (DIF), differential prediction and selection, employment testing, test validation, and classroom assessment.

Re-conceptualizing classroom assessment fairness: A systematic meta-ethnography of assessment literature and beyond

“…The emergent themes from our review of the assessment oriented studies suggest that fairness has been conceptualized in relation to summative (i.e., graded) and formative assessment practices. In the summative view, fairness has typically been characterized as including principles that transpire before, during, and after assessment administration, and comprise issues such as equal access to opportunity for learning, clear and consistent grading criteria, avoidance of score pollution, and accommodation based on students' ability and linguistic backgrounds (Abedi et al, 2004;Aitken, 2012;Camilli, 2006Camilli, , 2013Gipps, 1995;Gipps & Stobart, 2009;Klenowski, 2014;Scott et al, 2014;Suskie, 2000;Volante, 2006). Recent scholarship on fairness for formative assessment takes into account additional factors that have been shown to have a bearing on learning.…”

“…Nonetheless, studies that focused on accommodation for these two groups were included because accommodation has previously been emphasized as a fairness issue in the CA standards (DeLuca et al, 2016). Second, since the nature of CA fairness is different from fairness in large-scale traditions (Camilli, 2013;Tierney, 2014), studies (n = 128) that examined fairness in standardized testing contexts were excluded.…”

“…Compared to opportunity to learn, which typically considers fairness prior to assessment and which is closely aligned with instruction, the theme of access to demonstrate learningproviding students with "multiple, varied, equitable, and meaningful opportunities to demonstrate their learning" (Tierney, 2016, p.6)-has typically been characterized in terms of assessment opportunities that students are provided with to demonstrate what they know or can do (Camilli, 2006(Camilli, , 2013Cowie, 2015;Duffield & Spencer, 2002;Murillo & Hidalgo, 2017;Suskie, 2000;Tierney, 2013;Volante, 2006). The empirical studies have also demonstrated that students feel treated fairly when provided with multiple assessment opportunities (Duffield & Spencer, 2002;Mauldin, 2009).…”

An Evaluative Framework for Reviewing Fairness Standards and Practices in Educational Tests

“…For example, most standardization samples reported race/ethnicity using the U.S. Census Bureau's categories of White, African American, Hispanic, and Other. Narrow and limited definitions of subgroups that do not represent heterogeneity within groups is identified as an issue of inexact group classification and can occur when individuals are classified “by social address rather than educational histories” (Camilli, , p. 9). Sotel‐Dynega, Ortiz, Flanagan, and Chaplin () suggest that sample inclusion alone is insufficient to address fairness issues for ELLs and recommend defining factors that are more likely to be associated with performance differences such as length of language exposure and developmental language proficiency.…”

Examining Cultural Responsiveness in Large-Scale Assessment: The Matrix of Evidence for Validity Argumentation

“…[3, 5]. The inclusion of students from different cultural groups in large-scale assessment programs is often predicated upon the availability of analytical techniques for detecting cultural bias (see van de Vijver, 2016), including those based on item response theory (Camilli, 2013). However, because item bias detection is costly and requires the use of data obtained after administering tests to large samples of students, it is difficult or unlikely for this kind of scrutiny to take place in large-scale assessment programs, FIGURE 2 | Matrix of evidence for validity argumentation with a specific set of procedural assumptions and a specific set of assessment process components.…”