On Human Predictions with Explanations and Predictions of Machine Learning Models

Abstract

Humans are the final decision makers in critical tasks that involve ethical and legal concerns, ranging from recidivism prediction, to medical diagnosis, to fighting against fake news. Although machine learning models can sometimes achieve impressive performance in these tasks, these tasks are not amenable to full automation. To realize the potential of machine learning for improving human decisions, it is important to understand how assistance from machine learning models affects human performance and human agency.In this paper, we use deception detection as a testbed and investigate how we can harness explanations and predictions of machine learning models to improve human performance while retaining human agency. We propose a spectrum between full human agency and full automation, and develop varying levels of machine assistance along the spectrum that gradually increase the influence of machine predictions. We find that without showing predicted labels, explanations alone slightly improve human performance in the end task. In comparison, human performance is greatly improved by showing predicted labels (>20% relative improvement) and can be further improved by explicitly suggesting strong machine performance. Interestingly, when predicted labels are shown, explanations of machine predictions induce a similar level of accuracy as an explicit statement of strong machine performance. Our results demonstrate a tradeoff between human performance and human agency and show that explanations of machine predictions can moderate this tradeoff. CCS CONCEPTS• Applied computing → Law, social and behavioral sciences.

Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making

“…Multiple studies examined the effect of accuracy information [17,30,32], and found people to increase their trust in the model when high accuracy indicators are displayed, reflected both in subjective reporting and more consistent choices with the model's recommendations. Closest to ours is the work by Lai and Tan [17], where they studied the effect of showing prediction (in contrast to baseline without AI assistance), accuracy and multiple types of explanation for AI assisted decision-making in a deception-detection scenario. They found that all these features increased people's trust, measured as acceptance of the AI's recommendation as the final decision, and also the decision accuracy.…”

“…This chance number was calculated from the training dataset based on the percentages of people with the corresponding attribute-value earning income above 50K. We multiplied the percentages by 10 and rounded the number since prior work shows that people understand frequencies better than probabilities [17]. For example, in Figure 1, the chance value for occupation indicates that 5 people out of 10 with the occupation of Executive & Managerial have annual income above 50K.…”

“…We emphasize that this goal is distinct from enhancing trust in AI. For example, while research repeatedly demonstrates that providing high-performance indicators of an AI system, such as showing high accuracy scores, could enhance people's trust and acceptance of the system [17,30,32], they may not help people distinguish cases they can trust from those they should not. Meanwhile, ML is probabilistic and the probability of each single prediction can be indicated by a confidence score.…”

“…The main difference between the two measures was that in the with-prediction conditions, the agreement percentage would count the trials in which the participant's and the AI's predictions agreed and automatically counted as the final decision; whereas the switch percentage would only consider cases where they disagreed and had to make an intentional act of switching. Therefore, we consider switch percentage to be a stricter measure of trust, even though agreement percentage was used in prior research [17]. Figure 2 shows the switch percentage across the prediction and confidence factors.…”

Evaluating saliency map explanations for convolutional neural networks

“…Lai and Tan [35] demonstrated a trade-o between performance and human agency by exposing participants to varying levels of machine assistance (of an SVM) while they were identifying deceptive reviews. ey found that explanations without the suggestion of a label slightly improved human performance.…”