"Hello AI": Uncovering the Onboarding Needs of Medical Practitioners for Human-AI Collaborative Decision-Making

Abstract

Although rapid advances in machine learning have made it increasingly applicable to expert decision-making, the delivery of accurate algorithmic predictions alone is insufficient for effective human-AI collaboration. In this work, we investigate the key types of information medical experts desire when they are first introduced to a diagnostic AI assistant. In a qualitative lab study, we interviewed 21 pathologists before, during, and after being presented deep neural network (DNN) predictions for prostate cancer diagnosis, to learn the types of information that they desired about the AI assistant. Our findings reveal that, far beyond understanding the local, case-specific reasoning behind any model decision, clinicians desired upfront information about basic, global properties of the model, such as its known strengths and limitations, its subjective point-of-view, and its overall design objective-what it's designed to be optimized for. Participants compared these information needs to the collaborative mental models they develop of their medical colleagues when seeking a second opinion: the medical perspectives and standards that those colleagues embody, and the compatibility of those perspectives with their own diagnostic patterns. These findings broaden and enrich discussions surrounding AI transparency for collaborative decision-making, providing a richer understanding of what experts find important in their introduction to AI assistants before integrating them into routine practice. CCS Concepts: • Human-centered computing → Human computer interaction (HCI).

Shifting Concepts of Value

“…Research in HCI, and in particular Participatory Design (PD), has focused on shaping emergent technologies to support the practices and needs of those whose job it is to accomplish a given task or activity [13,14]. Recent studies in this area consider "acceptance" of algorithmic decision-support systems by practitioners [15]. Researchers looked at what information the practitioners desired when asked to make sense of an algorithmic diagnostic agent for decision-support in cancer diagnostics.…”

“…In the Cai et al [15] study, when asked about their desires the medical practitioners negotiated metrics of value and success for the diagnostic algorithmic agent. Some of these metrics would also apply to a human and follow the same logic as performance metrics.…”

“…Scholarship in this area thus far has produced many insights. Algorithmic decision-support systems are considered in safety-critical domains such as diagnostics and critical care [15], crisis management in due time [39], and containment of disease [27]. Yet, the value of algorithms for decision-support is disputed across domains.…”

Understanding Physician’s Experience With Conversational Interfaces During Occupational Health Consultation

“…This type of research addresses several critical design issues using, among others, more explainable AI frameworks [12], more precise data visualizations [13], and better presentation of information [27]. The second strand of research draws on the clinical appropriation of CDSTs to investigate the system as assistive and collaborative to the clinical routine [28]. Some early studies have explored HCI design strategies to fit intelligent systems into various clinical tasks.…”

“…Our results revealed that a chatbot with a passive type of interaction could be more efficient in assisting occupational health (OH) decision-making tasks. Some recent studies suggested that CDSTs should be designed to be unobtrusive [9], assistive [11], and collaborative [28]. Similarly, our CI mechanism with a passive assistant only provided on-demand feedback and lightweight information to assist with the consult without overburdening the concurrent tasks.…”