Are Explanations Helpful? A Comparative Study of the Effects of Explanations in AI-Assisted Decision-Making

Abstract

This paper contributes to the growing literature in empirical evaluation of explainable AI (XAI) methods by presenting a comparison on the effects of a set of established XAI methods in AI-assisted decision making. Specifically, based on our review of previous literature, we highlight three desirable properties that ideal AI explanations should satisfy-improve people's understanding of the AI model, help people recognize the model uncertainty, and support people's calibrated trust in the model. Through randomized controlled experiments, we evaluate whether four types of common model-agnostic explainable AI methods satisfy these properties on two types of decision making tasks where people perceive themselves as having different levels of domain expertise in (i.e., recidivism prediction and forest cover prediction). Our results show that the effects of AI explanations are largely different on decision making tasks where people have varying levels of domain expertise in, and many AI explanations do not satisfy any of the desirable properties for tasks that people have little domain expertise in. Further, for decision making tasks that people are more knowledgeable, feature contribution explanation is shown to satisfy more desiderata of AI explanations, while the explanation that is considered to resemble how human explain decisions (i.e., counterfactual explanation) does not seem to improve calibrated trust. We conclude by discussing the implications of our study for improving the design of XAI methods to better support human decision making. CCS CONCEPTS• Human-centered computing → Empirical studies in HCI; • Computing methodologies → Machine learning.

Machine Explanations and Human Understanding

“…Humans are typically asked to simulate model predictions given an input and some explanations [2, 9, 14, 15, 18, 20, 25, 35, 37-39, 45, 46, 50, 52, 58]. For example, given profiles of criminal defendants and machine explanations, participants are asked to guess what the AI model would predict [58].…”

“…Measuring human understanding of model decision boundary via feature importance. Additionally, Wang and Yin [58] also tested human understanding of model decision boundary via feature importance, specifically by (1) asking the participants to select among a list of features which one was most/least influential on the model's predictions and (2) specifying a feature's marginal effect on predictions. Ribeiro et al [51] asked participants to perform feature engineering by identifying features to remove, given the LIME explanations.…”

“…In those experiments, participants are shown machine predictions and explanations, then they are asked to give a final decision based on the information, with the goal of achieving complementary performance. For example, human decision-makers are asked to predict whether this defendant would re-offend within two years, given a machine prediction and explanations [58].…”

“…In contrast, Lai and Tan [34] show that feature importance improves human performance slightly in deceptive review detection, but human-AI teams fail to outperform AI alone. In fact, this is just one example of many papers that fail to improve the performance of human-AI teams in a wide range of tasks, including recidivism prediction, deceptive review detection, and hypoxemia prediction [6,12,22,23,28,33,41,50,58,59,61]. An intriguing puzzle is thus what factors drive such mixed results and how we can derive generalizable insights from empirical evaluations of explanations.…”

Better Together? An Evaluation of AI-Supported Code Translation

“…[37, p. 20]. Wang and Yin [95] examined the role of explanations in AI-supported decision making and found that various XAI methods were ineffective in supporting human decision makers on tasks for which they had limited domain expertise. Therefore, it is not always the case that the support provided by an AI model improves human decision-making quality.…”