It’s Complicated: The Relationship between User Trust, Model Accuracy and Explanations in AI

Abstract

Automated decision-making systems become increasingly powerful due to higher model complexity. While powerful in prediction accuracy, Deep Learning models are black boxes by nature, preventing users from making informed judgments about the correctness and fairness of such an automated system. Explanations have been proposed as a general remedy to the black box problem. However, it remains unclear if effects of explanations on user trust generalise over varying accuracy levels. In an online user study with 959 participants, we examined the practical consequences of adding explanations for user trust: We evaluated trust for three explanation types on three classifiers of varying accuracy. We find that the influence of our explanations on trust differs depending on the classifier’s accuracy. Thus, the interplay between trust and explanations is more complex than previously reported. Our findings also reveal discrepancies between self-reported and behavioural trust, showing that the choice of trust measure impacts the results.

It seems smart, but it acts stupid. Development of trust in AI advice in a repeated legal decision-making task

“…Yu et al [73] manipulated system accuracy on four levels (70%, 80%, 90%, and 100%), giving false positive/negative advice accordingly: trust decreased only in the 70 percent condition. Papenmeier et al [58] compared trust levels of participants that interacted with a high, medium, and low ("antagonistic") accuracy and found that participants indeed showed adequate levels of trust, in line with Yu and colleagues. To summarize, individuals can distinguish between appropriate and poor AI advice based on its accuracy, which is a crucial prerequisite for optimal trust calibration.…”

“…Although the majority of studies found positive effects of AI explanations on trust, some studies observed feelings of manipulation by AI [9] or subsequent overconfidence in AI [69]. Papenmeier et al [57,58] find evidence that not all explanations are helpful, and some might even be harmful: they discovered that adding nonsensical or random explanations hurt trust (as one would hope). Furthermore, their results show that explanations do not improve trust when individuals interact with a sufficiently accurate system.…”

“…With the underlying study, we follow up on previous work that has examined how different explanation types and performance levels affect trust [47,58]. We do so by analyzing trust on the basis of an applied scenario (the assessment of jail times), emphasizing trust development over time.…”

Effects of reliability indicators on usage, acceptance and preference of predictive process management decision support systems

“…There have been many approaches towards the definition and measurement of trust [46,63,82,112]. Trust is seen as a construct that is relevant both for relationships among humans and between humans and machines.…”

Visually Explaining Uncertain Price Predictions in Agrifood: A User-Centred Case-Study

“…First, there is no widely accepted definition for trust in intelligent systems, although many definitions have been proposed [72][73][74]. Second, measuring trust is very challenging because it evolves [75][76][77] and is affected by many factors [78], for example, domain expertise [75,77], visualised information and uncertainty [48,79], model accuracy [80,81], and level of transparency [82]. In addition, there is growing consensus among XAI researchers that optimising trust is not always desirable; rather, the stress should lie on appropriate trust [58] and trust calibration [83,84].…”