Algorithmic Justice in Child Protection: Statistical Fairness, Social Justice and the Implications for Practice

Abstract

Algorithmic tools are increasingly used in child protection decision-making. Fairness considerations of algorithmic tools usually focus on statistical fairness, but there are broader justice implications relating to the data used to construct source databases, and how algorithms are incorporated into complex sociotechnical decision-making contexts. This article explores how data that inform child protection algorithms are produced and relates this production to both traditional notions of statistical fairness and broader justice concepts. Predictive tools have a number of challenging problems in the child protection context, as the data that predictive tools draw on do not represent child abuse incidence across the population and child abuse itself is difficult to define, making key decisions that become data variable and subjective. Algorithms using these data have distorted feedback loops and can contain inequalities and biases. The challenge to justice concepts is that individual and group rights to non-discrimination become threatened as the algorithm itself becomes skewed, leading to inaccurate risk predictions drawing on spurious correlations. The right to be treated as an individual is threatened when statistical risk is based on a group categorisation, and the rights of families to understand and participate in the decisions made about them is difficult when they have not consented to data linkage, and the function of the algorithm is obscured by its complexity. The use of uninterpretable algorithmic tools may create ‘moral crumple zones’, where practitioners are held responsible for decisions even when they are partially determined by an algorithm. Many of these criticisms can also be levelled at human decision makers in the child protection system, but the reification of these processes within algorithms render their articulation even more difficult, and can diminish other important relational and ethical aims of social work practice.

Closing the AI accountability gap

“…The first (hypothetical) client wishes to develop a child abuse screening tool similar to that of the real cases extensively studied and reported on [11,14,15,21,25,36]. This complex case intersects heavily with applications in high-risk scenarios with dire consequences.…”

The agency of algorithms: Understanding human-algorithm interaction in administrative decision-making

“…For instance, Binns (2019: 19) notes that human decision-makers exercise discretion in the use of algorithms according to their own convictions and commitments. Keddell (2019) finds that child protection professionals in New Zealand have reservations regarding working with predictive algorithms they are unable to explain to families, while at the same time they are being held accountable for decisions based on them. This can lead to what Elish (2019) calls 'moral crumple zones', where human actors bear the brunt of malfunctions in automated decision-making procedures they have little or no control over.…”

Who is analysing what? The opportunities, risks and implications of using predictive risk modelling with Indigenous Australians in child protection: A scoping review

“…For complex decision contexts, such as child protection, where many interrelated factors need to be considered, predictive risk algorithms – including those that can condense vast amounts of information into a single risk score for decision support purposes – promise several advantages. The ability of these algorithms to process large amounts of data in short periods of time, their consistency in variable selection procedures and their adaptability to changing relationships in the data are appealing characteristics that are emphasised by proponents of PRMs (Cuccaro‐Alamin et al, 2017; Chouldechova et al, 2018; Keddell, 2019; Munro, 2019). Proponents of PRMs also assert that they can be used to mitigate bias in human decisions and to improve the accuracy of child protection workers’ decision‐making processes (Chouldechova et al, 2018).…”

“…A general limitation of their application in the child welfare context is that PRMs require high‐quality administrative data to provide accurate predictions and often rely on known instances of child abuse or neglect, which do not accurately measure the incidence of child abuse or neglect in the population at large (Eubanks, 2018). PRMs in child welfare systems have been viewed as particularly challenging due to the limitations of available data and the historical, political, legislative and cultural contexts in which child protection systems are embedded (Keddell, 2019; Munro, 2019; Saxena et al, 2020). These entrenched complexities within child protection systems are likely to present dangers for the application of PRMs as data collected by child protection systems are often measured with error, and subject to bias resulting from historic and systemic discrimination of marginalised groups and ethnic minorities (Munro, 2019), including Indigenous families.…”

“…This highlights the potential limitations of addressing biases through linking different administrative data sources. In settings such as these, PRMs may not mitigate existing biases in the system and may even exacerbate the risk of discrimination for vulnerable subpopulations (Keddell, 2019; Munro, 2019). Of note is that Indigenous overrepresentation is a crisis not unique to the Australian context: Black and Indigenous peoples are also overrepresented in North American and New Zealand (NZ) child protection and OOHC contexts (Boatswain‐Kyte et al, 2020; Cram et al, 2015; Fitzmaurice, 2020; Lawler et al, 2012; Roberts, 2014; Trocmé et al, 2004).…”