The Response Shift Paradigm to Quantify Human Trust in AI Recommendations

Abstract

Explainability, interpretability and how much they affect human trust in AI systems are ultimately problems of human cognition as much as machine learning, yet the effectiveness of AI recommendations and the trust afforded by end-users are typically not evaluated quantitatively. We developed and validated a general purpose Human-AI interaction paradigm which quantifies the impact of AI recommendations on human decisions. In our paradigm we confronted human users with quantitative prediction tasks: asking them for a first response, before confronting them with an AI's recommendations (and explanation), and then asking the human user to provide an updated final response. The difference between final and first responses constitutes the shift or sway in the human decision which we use as metric of the AI's recommendation impact on the human, representing the trust they place on the AI. We evaluated this paradigm on hundreds of users through Amazon Mechanical Turk using a multi-branched experiment confronting users with good/poor AI systems that had good, poor or no explainability. Our proof-ofprinciple paradigm allows one to quantitatively compare the rapidly growing set of XAI/IAI approaches in terms of their effect on the end-user and opens up the possibility of (machine) learning trust.

Explanation matters: An experimental study on explainable AI

“…Based on the fact that the literature to date has produced mixed results, we concluded that context-specific (confounding) factors have been overlooked so far that may explain why some studies find significant results and others report non-significant relationships. One reason could be the actual or the perceived performance of the AI (Shafti et al, 2022). Another factor could be the type of explanation (e.g., David et al, 2021;Lai et al, 2020;van der Waa et al, 2021) that leads participants to follow the system's advice more often.…”

“…The participants' task in the study was to identify hateful content via the user interface to detect hate speech. Shafti et al (2022) observed that good explanations of XAI can lead to a significantly lower error rate, a higher human performance and higher user confidence in AI. In their experimental study, a grade prediction task of students was used based on tabular data about the student's background (e.g., parents' jobs or weekly study time).…”

“…In summary, research on XAI is flourishing and there are several studies highlighting a positive impact of XAI on outcome variables such as trust (David et al, 2021;Druce et al, 2021;Shafti et al, 2022;Weitz et al, 2019Weitz et al, , 2021 or human performance (Lai et al, 2020;Ray et al, 2019;Shafti et al, 2022). However, there are also studies showing that the effectiveness of XAI is related to specific implementations (e.g., van der Waa et al, 2021) and that XAI does not per se lead to intended effects (e.g., Alufaisan et al, 2020).…”

Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment

“…Particularly, end users’ decision to act on or dismiss AI recommendations may be attached to some human-centred AI design characteristics and the degree of AI explainability. 21 22 Human factor aspects are central in AI-based decision support systems in safety critical applications, 23 prompting us to keep actively engineering safety into AI systems.…”

Uncertainty in XAI: Human Perception and Modeling Approaches

“…There are a number of studies that report positive effects of explanations on human perception [23,[41][42][43][44][45][46][47]. These experiments investigated tasks in various modalities and ranged from asking users to detect hate speech, predict students' grades, predict an agent's performance in frozen frames of video games, or decide on (virtual) patients' insulin intake.…”