How to Train a (Bad) Algorithmic Caseworker: A Quantitative Deconstruction of Risk Assessments in Child Welfare

Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders

“…There are practice models... that help [workers] explore some very concrete, specific questions that help force them not to just make decisions based on their own hunch. " However, other participants said these low-tech tools had built-in biases (see [116]) and workers frequently manipulate them (against their training) to produce any desired output (P1,P5,P6,P7,P9,P15,P16). 19 Some said diagnostic checklists could be used better if workers were better trained and held accountable to follow the training (P5,P10,P14).…”

Algorithms in the Daily Lives of Child-Welfare Caseworkers: Harms to Practice, Agency, and Street-Level Decision-Making

“…1 Studies have found that value conflicts arise when the logics embedded within the government's digital platforms do not align with street-level bureaucrats' discretion when they tried enacting the same shared values in practice [37,61,66,126]. Research within public administration, science and technology studies, and human-computer interaction have recently drawn attention to how the digitization of public services is leading to distinct changes in street-level bureaucrats' discretion [21], power asymmetries between public officials and citizens [106,107,114], need for re-skilling of public officials [8,61,66], and actor transparency in government decision-making [48]. Lindgren et al [73] go a step further and argue that "public officials can no longer be understood as merely human".…”

“…They predict "decreased bargaining and discretionary power of governmental workers" as one of the early warning signs of this phenomenon. These changes in professional work practices through the adoption of digital tools have similar, yet more serious implications for the adoption of algorithmic tools which further shift discretion away from public service workers; instances of which are already being captured by recent studies in the public sector [66,105].…”

“…They show how families are labeled "risky by association" because designers aggregated risk scores at the household level and how systemic discrimination is amplified by the inclusion of static variables that capture families' involvement with the criminal legal system or public benefits programs. Similarly, Saxena et al [105] draw attention to the multiplicity and temporality of risk factors that are experienced by families in the system, how the system itself poses significant risk where families experience over-surveillance and are subjected to several arbitrary decisions, and how these street-level structural biases become embedded in algorithmic systems such as the Allegheny Family Screening Tool.…”

“…Algorithms are being deployed across the public sector where they are informing decisions regarding which child maltreatment cases to investigate [66,101], which families should be offered preventive services [106], which neighborhoods require more policing [17,55], and who are offered public housing and unemployment benefits [43,70], among others. However, audits of these systems show that they are achieving worse outcomes [29,97], embedding human biases in decision-making [107], and exacerbating racial and socio-economic biases [43,68,94]. Algorithmic systems are causing real harm to vulnerable people who are witnessing cuts to their unemployment and healthcare benefits [36,43], loss of public housing [125], as well as experiencing unfair investigations from the child welfare system [59,114].…”