Putting explainable AI in context: institutional explanations for medical AI

Abstract

There is a current debate about if, and in what sense, machine learning systems used in the medical context need to be explainable. Those arguing in favor contend these systems require post hoc explanations for each individual decision to increase trust and ensure accurate diagnoses. Those arguing against suggest the high accuracy and reliability of the systems is sufficient for providing epistemic justified beliefs without the need for explaining each individual decision. But, as we show, both solutions have limitations—and it is unclear either address the epistemic worries of the medical professionals using these systems. We argue these systems do require an explanation, but an institutional explanation. These types of explanations provide the reasons why the medical professional should rely on the system in practice—that is, they focus on trying to address the epistemic concerns of those using the system in specific contexts and specific occasions. But ensuring that these institutional explanations are fit for purpose means ensuring the institutions designing and deploying these systems are transparent about the assumptions baked into the system. This requires coordination with experts and end-users concerning how it will function in the field, the metrics used to evaluate its accuracy, and the procedures for auditing the system to prevent biases and failures from going unaddressed. We contend this broader explanation is necessary for either post hoc explanations or accuracy scores to be epistemically meaningful to the medical professional, making it possible for them to rely on these systems as effective and useful tools in their practices.

Bias in AI-based models for medical applications: challenges and mitigation strategies

“…The question of whether AI algorithms may need to be more generalizable, trained on larger and more diverse datasets to be applied to broader populations, or more localized and applied narrowly remains to be addressed. In any case, AI models will have to be explainable 15 with transparent methodologies so that these questions can be studied and debated in the coming years.…”

Acceptance and understanding of artificial intelligence in medical research among orthopaedic surgeons

“…If the surgeon is not able to explain the basis on which they make their decisions, they are therefore unable to justify them. 26 That negative thoughts were moderated by positive ones in theme four may be due to the rise of AI in areas of modern life outside medical research. 20 Modern society is firmly in the information age -a shift the size of which has been likened to the Industrial Revolution.…”

Artificial intelligence and heart failure: A state‐of‐the‐art review

“…Moreover, explainable AI can help clinicians understand how AI algorithms make decisions and how they arrive at their recommendations. 85 Regulatory approval and certification can help further establish clinician's trust in the safety and effectiveness of AI algorithms. 86 Clinicians will be more likely to trust AI algorithms that have been approved by regulatory bodies such as the FDA, and that have undergone rigorous certification processes.…”

“…For this, algorithms should undergo rigorous testing and validation to ensure that they perform as intended across a range of scenarios and patient populations. Moreover, explainable AI can help clinicians understand how AI algorithms make decisions and how they arrive at their recommendations 85 . Regulatory approval and certification can help further establish clinician's trust in the safety and effectiveness of AI algorithms 86 .…”