Explainable recommendation: when design meets trust calibration

Abstract

Human-AI collaborative decision-making tools are being increasingly applied in critical domains such as healthcare. However, these tools are often seen as closed and intransparent for human decision-makers. An essential requirement for their success is the ability to provide explanations about themselves that are understandable and meaningful to the users. While explanations generally have positive connotations, studies showed that the assumption behind users interacting and engaging with these explanations could introduce trust calibration errors such as facilitating irrational or less thoughtful agreement or disagreement with the AI recommendation. In this paper, we explore how to help trust calibration through explanation interaction design. Our research method included two main phases. We first conducted a think-aloud study with 16 participants aiming to reveal main trust calibration errors concerning explainability in AI-Human collaborative decision-making tools. Then, we conducted two co-design sessions with eight participants to identify design principles and techniques for explanations that help trust calibration. As a conclusion of our research, we provide five design principles: Design for engagement, challenging habitual actions, attention guidance, friction and support training and learning. Our findings are meant to pave the way towards a more integrated framework for designing explanations with trust calibration as a primary goal.

Effects of reliability indicators on usage, acceptance and preference of predictive process management decision support systems

“…However, also interdisciplinary approaches have been presented that incorporate cognitive psychology and philosophy in order to reframe and reflect the motivations and reasons why explanations are to be used in intelligent systems [54]. Recently, there have been approaches to design for explainability of advanced machine learning to non-expert users [15,27,75]. So far, the learnings have not been transferred to specific application areas, such as construction.…”

“…Interestingly, the analysis by Zhang et al [114] represents only one of few studies that consider the impact on domain experts rather than endusers. For clinical decision support systems, [75] found that explanations did not always effectively support users in calibrating their trust, due to conflicts with usability and required efforts.…”

How the different explanation classes impact trust calibration: The case of clinical decision support systems

“…We also aim at broadening discussions on explainable AI for collaborative decision-making and paving the way for more research on how to customise and contextualise explanations so that they fit users' needs and expectations of each of their different classes. Compared to our previous work (Naiseh et al, 2021a;Naiseh et al, 2021b), this study adds a quantitative evaluation of different XAI classes and their effect on users' trust calibration. It also elicits XAI interface requirements correlated with particular XAI classes.…”

“…We first explored what errors users make while interacting with XAI interfaces (Naiseh et al, 2021b). We then conducted an explanatory study to discover potential design solutions to mitigate users' errors (Naiseh et al, 2021a).…”

Explanation-by-Example Based on Item Response Theory

“…Developed by AI and Human Computer Interaction (HCI) researchers, XAI is a user-centric field of study aimed at developing techniques to make the functioning of these systems and models more transparentand consequently more reliable [2]. Recent research shows that the trust calibration on the models' decision is very important, since exaggerated or measured confidence can lead to critical problems depending on the context [19].…”