Measurements, Algorithms, and Presentations of Reality: Framing Interactions with AI-Enabled Decision Support

Abstract

Bringing AI technology into clinical practice has proved challenging for system designers and medical professionals alike. The academic literature has, for example, highlighted the dangers of black-box decision-making and biased datasets. Further, end-users’ ability to validate a system’s performance often disappears following the introduction of AI decision-making. We present the MAP model to understand and describe the three stages through which medical observations are interpreted and handled by AI systems. These stages are Measurement, in which information is gathered and converted into data points that can be stored and processed; Algorithm, in which computational processes transform the collected data; and Presentation, where information is returned to the user for interpretation. For each stage, we highlight possible challenges that need to be overcome to develop Human-Centred AI systems. We illuminate our MAP model through complementary case studies on colonoscopy practice and dementia diagnosis, providing examples of the challenges encountered in real-world settings. By defining Human-AI interaction across these three stages, we untangle some of the inherent complexities in designing AI technology for clinical decision-making, and aim to overcome misalignment between medical end-users and AI researchers and developers.

“If I Had All the Time in the World”: Ophthalmologists’ Perceptions of Anchoring Bias Mitigation in Clinical AI Support

“…Specically, they wished for the AI system to produce fewer false positive alerts and be able to detect more types of abnormalities. This points to a mismatch between clinicians' mental model and expectations of the AI's capabilities [62]. Similar observations were made by Yang et al in their investigation of decision-making practices in the context of heart pump implants [70].…”

“…Amidst an increase in the interest and use of AI-driven decision support tools, cognitive biases due to these systems have become an area of concern within HCI [14,42,62,67,71]. Here, we discuss related work across three key topics.…”

“…Given the widespread occurrence of cognitive biases [60] and the eect of task context on the suitability of specic bias mitigation strategies [62], the literature has explored a variety of methods and techniques for overcoming these biases. The use of educational strategies for recognising and subsequently addressing cognitive biases has often been suggested, although the eect of these strategies has been inconclusive [41].…”