“Why Do I Care What’s Similar?” Probing Challenges in AI-Assisted Child Welfare Decision-Making through Worker-AI Interface Design Concepts

Abstract

Data-driven AI systems are increasingly used to augment human decision-making in complex, social contexts, such as social work or legal practice. Yet, most existing design knowledge regarding how to best support AI-augmented decision-making comes from studies in comparatively well-defned settings. In this paper, we present fndings from design interviews with 12 social workers who use an algorithmic decision support tool (ADS) to assist their day-to-day child maltreatment screening decisions. We generated a range of design concepts, each envisioning diferent ways of redesigning or augmenting the ADS interface. Overall, workers desired ways to understand the risk score and incorporate contextual knowledge, which move beyond existing notions of AI interpretability. Conversations around our design concepts also surfaced more fundamental concerns around the assumptions underlying statistical prediction, such as inference based on similar historical cases and statistical notions of uncertainty. Based on our fndings, we discuss how ADS may be better designed to support the roles of human decision-makers in social decision-making contexts. CCS CONCEPTS• Human-centered computing → Human Computer Interaction (HCI); Interactive system and tools.

Toward Supporting Perceptual Complementarity in Human-AI Collaboration via Reflection on Unobservables

“…Yet in many real-world settings where AI-based decision support tools are used, the total number of model-observable features and relevant unobservables is significantly larger. For example, call workers tasked with screening child maltreatment calls have hundreds of features available to them [8,28]. In settings where a larger number of features are present, it is possible that highlighting the pieces of information that are not available to an AI model would have a greater effect in improving their predictive performance.…”

“…Recent research has also demonstrated the potential of information asymmetry as a source of complementary human-AI performance, showing that humans can successfully integrate contextual information when making use of AI recommendations [5,11,21,22]. However, in many settings human decision-makers do not have a clear understanding of what information is and is not available to the algorithm [24,27,28], which may hinder their ability to integrate contextual information. Furthermore, little is known regarding how to best design ADS to support human decision-makers in effectively integrating across AI outputs and unobservables.…”

“…For example, in the context of AI-augmented healthcare, an AI system may have a greater ability to parse through large quantities of time series data collected via continuous bedside monitoring, while a physician can better perceive changes in patients' emotional state and listen to their lived experience [32,48]. Similarly, in the context of AI-augmented child welfare decision-making, AI models may have access to large quantities of administrative data, while humans have access to the rich information communicated during a phone conversation with a caller to a child abuse hotline [10,11,27,28]. More generally, consider ADS tools that rely on machine learned prediction models, which estimate an outcome or a probability of an event (e.g., a predicted house selling price or the risk of an adverse health outcome), given a set of covariates or features available to the model.…”

“…In many real-world settings, such as the examples described above, some of these features are available to the human, and it is the hope of those deploying ADS tools that the humans using these tools will be able to complement the AI model's capabilities by integrating this additional information into their decisions. However, prior field research indicates that humans are often unaware of exactly which sources of information are uniquely available to themselves versus an AI model [24,27,28]. Little is known regarding how we might design ADS tools to amplify this form of human-AI complementarity.…”

“…Instead, call workers were more likely to avoid errors of omission, which correspond to false negative errors, by integrating other sources of information instead of relying on AI outputs alone [11]. However, a series of interview studies with call screeners revealed that they felt uncertain about which information the AI model did or did not have access to, and that they wanted more support from the ADS interface in understanding how exactly this information complemented other information sources at their disposal [27,28].…”