Explainable machine learning practices: opening another black box for reliable medical AI

Abstract

In the past few years, machine learning (ML) tools have been implemented with success in the medical context. However, several practitioners have raised concerns about the lack of transparency—at the algorithmic level—of many of these tools; and solutions from the field of explainable AI (XAI) have been seen as a way to open the ‘black box’ and make the tools more trustworthy. Recently, Alex London has argued that in the medical context we do not need machine learning tools to be interpretable at the algorithmic level to make them trustworthy, as long as they meet some strict empirical desiderata. In this paper, we analyse and develop London’s position. In particular, we make two claims. First, we claim that London’s solution to the problem of trust can potentially address another problem, which is how to evaluate the reliability of ML tools in medicine for regulatory purposes. Second, we claim that to deal with this problem, we need to develop London’s views by shifting the focus from the opacity of algorithmic details to the opacity of the way in which ML tools are trained and built. We claim that to regulate AI tools and evaluate their reliability, agencies need an explanation of how ML tools have been built, which requires documenting and justifying the technical choices that practitioners have made in designing such tools. This is because different algorithmic designs may lead to different outcomes, and to the realization of different purposes. However, given that technical choices underlying algorithmic design are shaped by value-laden considerations, opening the black box of the design process means also making transparent and motivating (technical and ethical) values and preferences behind such choices. Using tools from philosophy of technology and philosophy of science, we elaborate a framework showing how an explanation of the training processes of ML tools in medicine should look like.

More than meets the AI: The possibilities and limits of machine learning in olfaction

“…This article discusses the danger of the hidden assumptions present in current applications of machine learning (ML in the following) in olfaction. Our analysis extends previous critiques of ML-models in biological systems that cautioned of methodological pitfalls, including incomplete data and small datasets or false positives, opacity in algorithm design, and a lack of empirical and contextual grounding (Johnson, 2008;Coveney et al, 2016;Ratti and López-Rubio, 2018;London, 2019;Ratti and Graves, 2022). The scientifically important issue raised with this analysis is that "[these pitfalls do] not explain how we arrive at the wrong model, just how we accept the wrong model" (Johnson, 2008, 25, added emphasis).…”

What kind of trust does AI deserve, if any?

“…3. And finally, related to the point above, trust in AI is not like trust in pharmaceuticals, as some suggest (see [50,68]), because we do not trust medical artifacts such as pharmaceuticals in their capacity as conveyers of information but rather as capable of performing a chemical intervention in our bodies.…”

“…Another way to see that they are indeed distinct concepts is to simply think of the ways in which they often fail to pair, or map onto one another, in human behavior: we often trust things that are not trustworthy and fail to trust things that are Ferrario and Loi [27]. For those interested in a technical approach to trust, issues related to transparency and explainability are prominent [22,27,68] (Ribeiro et al 2016). Often, proponents of this approach draw from insights concerning trust in other technological domains such as computer simulations (Durán and Formanek [21]) [22]), IT systems in general [15], or autonomous technologies such as cars [18,38].…”

The ethics of AI-assisted warfighter enhancement research and experimentation: Historical perspectives and ethical challenges

“…Despite occasional optimism about rendering AI decision-making transparent (e.g., Mishra, 2021), most scholars remain concerned about the effects of biased AI used for medical purposes. Among these are concerns that biased AI will reduce persons to mere data (Sparrow and Hatherley, 2019), that AI might impermissibly (and invisibly) incorporate economic data in its rationing recommendations (Sparrow and Hatherley, 2020;Braun et al, 2021), and that AI will rely upon other value-laden considerations (Ratti and Graves, 2022). Again, this is merely a sampling of the technological risks associated with AI.…”