Algorithmic Bias in Education

Abstract

Draft Preprint. In this paper, we review algorithmic bias in education, discussing the causes of that bias and reviewing the empirical literature on the specific ways that algorithmic bias is known to have manifested in education. While other recent work has reviewed mathematical definitions of fairness and expanded algorithmic approaches to reducing bias, our review focuses instead on solidifying the current understanding of the concrete impacts of algorithmic bias in education—which groups are known to be impacted and which stages and agents in the development and deployment of educational algorithms are implicated. We discuss theoretical and formal perspectives on algorithmic bias, connect those perspectives to the machine learning pipeline, and review metrics for assessing bias. Next, we review the evidence around algorithmic bias in education, beginning with the most heavily-studied categories of race/ethnicity, gender, and nationality, and moving to the available evidence of bias for less-studied categories, such as socioeconomic status, disability, and military-connected status. Acknowledging the gaps in what has been studied, we propose a framework for moving from unknown bias to known bias and from fairness to equity. We discuss obstacles to addressing these challenges and propose four areas of effort for mitigating and resolving the problems of algorithmic bias in AIED systems and other educational technology.

The politics and reciprocal (re)configuration of accountability and fairness in data-driven education

“…The public demanded accountability after what can only be described as a fraught and messy policymaking process involving competing conceptions of fairness, unavoidable harms, and an insufficient appeal system (Adams, Weale, and Barr 2020;Kippin and Cairney 2021). Algorithmic bias raises concern in various application areas of AIED technologies, including educational assessment, students' dropout risk prediction, and algorithmic ability-grouping (see Baker & Hawn [2021]; Kizilcec & Lee [forthcoming]; Carmel & Ben-Shahar [2017]).…”

“…Trained on historical data, ML algorithms may infer (multiple) proxies for legally protected or otherwise sensitive attributes (e.g., 'race', gender, or socio-economic status), consequently introducing disparities into algorithmic predictions (Baker & Hawn [2021]; Mehrabi et al [2021]). Due to unrepresentative sampling and other technical design flaws, bias against specific groups in outcomes may occur without explicit use of protected or sensitive attributes as model features.…”

“…Fair ML metrics bear similarities to fairness measures for tests in education (see Camilli [2013]; Hutchinson & Mitchell [2019]) 1 , and they have been increasingly discussed in educational contexts (Kizilcec & Lee [forthcoming]; see also Baker & Hawn [2021]). They have a two-fold function: Firstly, designed to capture distinct philosophical and legal concepts of discrimination and equality, they function as idealized formal measures for fairness.…”

“…In practice, algorithms cannot be audited for fairness across all possible (sub)groups, meaning certain groups must be prioritized in selecting comparison classes for fairness analyses. Protected groups specified in non-discrimination legislation (e.g., 'race', gender) are common choices, although other factors (e.g., urbanicity, socio-economic status) may be accounted for (Baker and Hawn 2021). The results of fairness analyses depend on what categories are selected as well as on how they are constructede.g., how 'racial categories' are abstracted 'into a mathematically comparable form' (Hanna et al 2020, 508).…”

“…Data-driven education and educational artificial intelligence technologies (AIED) promise 'flexible, inclusive, personalized, engaging, and effective' teaching and learning (Luckin et al 2016, 18), and increased equity and accountability (see Fontaine [2016]). However, AIED technologies themselves can involve significant risks of algorithmic discrimination and unfairness in pedagogical and administrative practices with high-stakes and which are conducted in educational domains already characterized by patterned inequalities and structural discrimination (Baker & Hawn [2021]; Kizilcec & Lee [forthcoming]). Machine learning (ML) systems, which dig through troves of student and learning data, learn biases due to technical design errors but also by picking up on existing patterns of inequality and discrimination (Baker and Hawn 2021).…”