A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfare

Abstract

Algorithms have permeated throughout civil government and society, where they are being used to make high-stakes decisions about human lives. In this paper, we first develop a cohesive framework of algorithmic decision-making adapted for the public sector (ADMAPS) that reflects the complex socio-technical interactions between human discretion, bureaucratic processes, and algorithmic decision-making by synthesizing disparate bodies of work in the fields of Human-Computer Interaction (HCI), Science and Technology Studies (STS), and Public Administration (PA). We then applied the ADMAPS framework to conduct a qualitative analysis of an in-depth, eight-month ethnographic case study of algorithms in daily use within a child-welfare agency that serves approximately 900 families and 1300 children in the mid-western United States. Overall, we found that there is a need to focus on strength-based algorithmic outcomes centered in social ecological frameworks. In addition, algorithmic systems need to support existing bureaucratic processes and augment human discretion, rather than replace it. Finally, collective buy-in in algorithmic systems requires trust in the target outcomes at both the practitioner and bureaucratic levels. As a result of our study, we propose guidelines for the design of high-stakes algorithmic decision-making tools in the child-welfare system, and more generally, in the public sector. We empirically validate the theoretically derived ADMAPS framework to demonstrate how it can be useful for systematically making pragmatic decisions about the design of algorithms for the public sector.CCS Concepts: • Human-centered computing → Human-computer interaction (HCI); Empirical studies in HCI ; • Applied computing → Computing in government.

Unpacking Invisible Work Practices, Constraints, and Latent Power Relationships in Child Welfare through Casenote Analysis

“…We find support that computational text analysis of casenotes can be a powerful tool for developing holistic decision-support tools instead of the popular administrative data-centered risk assessment tools [36,44] that have been found to be biased [115]. This answers calls in prior SIGCHI research about the possibility of using case narratives as an important research tool [7,116].…”

“…SIGCHI scholars have thus begun studying data-driven practices that adopt care as a design lens to create systems that advocate for a caring democracy [64,94,122]. Despite two decades of adoption of digital technologies (often referred to as Digital Era Governance [26]) and promises of transformation, these tools have generally fused onto existing human discretionary practices rather than altering them at a deeper organizational level [71,115,125]. Digital technologies have raised the need to understand human discretionary work conducted by bureaucrats who must balance citizens' needs against the demands of policymakers as they acquire new skills and learn to make decisions through these systems [115].…”

“…As a result, recent HCI scholarly work has sought to unpack the nature of human discretionary work conducted at the street-level in public services [6,37,71,104,110,115]. Alkhatib and Bernstein introduced the theory of street-level algorithms to distinctly highlight the gaps in algorithmic decision-making that human discretion needed to address [6].…”

“…SIGCHI researchers have also used crowdsourcing platforms such as Amazon Mechanical Turk (MTurk) to study people's perceptions of algorithmic decisions and their impact on human judgment [66,86]. However, as highlighted by recent ethnographic work in CWS [21,115], there are drawbacks in these studies that need redressing: 1) algorithms built from quantitative administrative data in CWS only account for a narrow set of predictors, offering a deficit-based framing of families [116], and 2) experiments conducted on crowdsourcing platforms do not account for organizational/legislative constraints or day-to-day bureaucratic protocols that impact decision-making for all cases [115]. In light of these concerns, SIGCHI researchers have suggested that collaboratively curated caseworker documentation (i.e., caseworkers' narratives) may offer a more holistic picture of street-level interactions and bureaucratic complexities [7,115].…”

“…However, as highlighted by recent ethnographic work in CWS [21,115], there are drawbacks in these studies that need redressing: 1) algorithms built from quantitative administrative data in CWS only account for a narrow set of predictors, offering a deficit-based framing of families [116], and 2) experiments conducted on crowdsourcing platforms do not account for organizational/legislative constraints or day-to-day bureaucratic protocols that impact decision-making for all cases [115]. In light of these concerns, SIGCHI researchers have suggested that collaboratively curated caseworker documentation (i.e., caseworkers' narratives) may offer a more holistic picture of street-level interactions and bureaucratic complexities [7,115]. Unlike administrative quantitative data, caseworker narratives offer a more credible source of information by revealing workers' interactions with families, uncertainties in a case, and impact of bureaucratic constraints on decision-making.…”