Examining the effect of explanation on satisfaction and trust in AI diagnostic systems

Abstract

Background Artificial Intelligence has the potential to revolutionize healthcare, and it is increasingly being deployed to support and assist medical diagnosis. One potential application of AI is as the first point of contact for patients, replacing initial diagnoses prior to sending a patient to a specialist, allowing health care professionals to focus on more challenging and critical aspects of treatment. But for AI systems to succeed in this role, it will not be enough for them to merely provide accurate diagnoses and predictions. In addition, it will need to provide explanations (both to physicians and patients) about why the diagnoses are made. Without this, accurate and correct diagnoses and treatments might otherwise be ignored or rejected. Method It is important to evaluate the effectiveness of these explanations and understand the relative effectiveness of different kinds of explanations. In this paper, we examine this problem across two simulation experiments. For the first experiment, we tested a re-diagnosis scenario to understand the effect of local and global explanations. In a second simulation experiment, we implemented different forms of explanation in a similar diagnosis scenario. Results Results show that explanation helps improve satisfaction measures during the critical re-diagnosis period but had little effect before re-diagnosis (when initial treatment was taking place) or after (when an alternate diagnosis resolved the case successfully). Furthermore, initial “global” explanations about the process had no impact on immediate satisfaction but improved later judgments of understanding about the AI. Results of the second experiment show that visual and example-based explanations integrated with rationales had a significantly better impact on patient satisfaction and trust than no explanations, or with text-based rationales alone. As in Experiment 1, these explanations had their effect primarily on immediate measures of satisfaction during the re-diagnosis crisis, with little advantage prior to re-diagnosis or once the diagnosis was successfully resolved. Conclusion These two studies help us to draw several conclusions about how patient-facing explanatory diagnostic systems may succeed or fail. Based on these studies and the review of the literature, we will provide some design recommendations for the explanations offered for AI systems in the healthcare domain.

The Response Shift Paradigm to Quantify Human Trust in AI Recommendations

“…Self-report measures as a measure of trust are especially problematic, as they are well known to be affected by biases such as the perceived desirability of answering in a certain manner and are inconsistent between subjects, over-relying on the subject's perception of themselves (Paulhus et al, 2007). Trust in AI models has been studied previously for classification tasks such as disease diagnosis (Anton et al, 2022;Alam & Mueller, 2021). This is, however, not applicable to AI models that make predictions on a continuous scale.…”

Machine Learning and Knowledge Extraction

“…For instance, Zhou et al [13] showed that the explanation of influences of training data points on predictions significantly increased the user trust in predictions. Alam and Mueller [14] investigated the roles of explanations in AI-informed decision-making in medical diagnosis scenarios. The results show that visual and example-based explanations integrated with rationales had a significantly better impact on patient satisfaction and trust than no explanations, or with text-based rationales alone.…”

“…Furthermore, with the advancement of AI explanation research, different explanation approaches such as local and global explanations, as well as feature importance-based and example-based explanations are proposed [6]. As a result, besides the explanation presentation styles such as visualisation and text [14,15], it is also critical to understand how different explanation approaches affect user trust in AI-informed decision-making. In addition, Edwards [16] stated that the main challenge for AI-informed decision-making is to know whether an explanation that seems valid is accurate.…”

How Do We Assess the Trustworthiness of AI? Introducing the Trustworthiness Assessment Model (TrAM)

“…Consequently, PT answers the question "How trustworthy do I think the system is?". PT is another latent construct that is not directly observable but that can be measured indirectly as is commonly done in research assessing people's perceptions of system trustworthiness (e.g., by asking people to report on their perceived trustworthiness of a system [3,97], by observing people's interactions with systems [104,117]). PT is a trustor's assessment of the AT of the system based on a cognitive and affective evaluation of the system [6,70,77].…”