‘If You’re Going to Trust the Machine, Then That Trust Has Got to Be Based on Something’:

Abstract

The role of Artificial Intelligence (AI) in clinical decision-making raises issues of trust. One issue concerns the conditions of trusting the AI which tends to be based on validation. However, little attention has been given to how validation is formed, how comparisons come to be accepted, and how AI algorithms are trusted in decision-making. Drawing on interviews with collaborative researchers developing three AI technologies for the early diagnosis of pulmonary hypertension (PH), we show how validation of the AI is jointly produced so that trust in the algorithm is built up through the negotiation of criteria and terms of comparison during interactions. These processes build up interpretability and interrogation, and co-constitute trust in the technology. As they do so, it becomes difficult to sustain a strict distinction between artificial and human/social intelligence.

(De)troubling transparency: artificial intelligence (AI) for clinical applications

“…The study of the screening algorithm—which is the focus of this article—took place against the background of the broader study of all three applications. The broader study documenting the development of all three applications is described in ( Winter and Carusi 2022 ). Fieldwork included interviews and observations of the interactions and discussions, of ideas, scientific criteria and concepts that shape transparency in AI development.…”

“…We have already spoken about the ongoing ‘refinement’ of variables or criteria that play out in the continual movement between ‘data’ and ‘outputs’ in software building (see Winter and Carusi 2022 ). These refining or tinkering practices open up a space for building the software and training it together, and contextualises the codes so as to give traction and meaning to diagnostic codes and variables selected for the model.…”

“…In this article, we describe the development process of one AI application for a clinical setting, where the very initial steps are being made before full validation ( Winter and Carusi 2022 ). We show how epistemic transparency is negotiated and co-produced in the close collaboration between AI developers, clinicians and a biomedical scientist, forming the context in which AI is accepted as an epistemic operator.…”

“…This is an important consideration for our study since collaborators often highlighted these issues as part and parcel of the early development stage and necessary for development to proceed. At this early developmental stage, the algorithms just needed to be adequate for the work at hand (ie, as a proof of concept in the laboratory) subject to ongoing refinement (see Winter and Carusi 2022 ). This calls attention to a direction for further work which might study how his process of ‘staying with the trouble’ proceeds, and when (or if) it ever stops.…”

AI model transferability in healthcare: a sociotechnical perspective

“…Research demonstrates that model trust and use in single, local applications depends upon users' involvement in model development and on the social capital of clinical champions who legitimize the model 10,12 . However, social capital is often local and does not transfer across institutions, and it degrades rapidly when clinical champions leave an organization, endangering trust in models.…”