Show or Suppress? Managing Input Uncertainty in Machine Learning Model Explanations

Abstract

Feature attribution is widely used in interpretable machine learning to explain how influential each measured input feature value is for an output inference. However, measurements can be uncertain, and it is unclear how the awareness of input uncertainty can affect the trust in explanations. We propose and study two approaches to help users to manage their perception of uncertainty in a model explanation: 1) transparently show uncertainty in feature attributions to allow users to reflect on, and 2) suppress attribution to features with uncertain measurements and shift attribution to other features by regularizing with an uncertainty penalty. Through simulation experiments, qualitative interviews, and quantitative user evaluations, we identified the benefits of moderately suppressing attribution uncertainty, and concerns regarding showing attribution uncertainty. This work adds to the understanding of handling and communicating uncertainty for model interpretability.

The Role of Human Knowledge in Explainable AI

“…Finally, while explanations were proven to be effective in leading the user in achieving a task and improving their trust and understanding of the model, it has also been demonstrated that sometimes they are either not able to improve [97,98] or, worse, they reduce human accuracy and trust [99]. A similar result in a different context was found by Dinu et al [100].…”

The Effect of Interpretable Artificial Intelligence on Repeated Managerial Decision-Making Under Uncertainty

“…They find that users resist advice from an algorithm more when the environment is more uncertain. D. Wang et al (2021) find that communicating the level of uncertainty to users does not affect trust, confidence, or decision-making quality, but users spend more time making decisions when they receive information about uncertainty. In their work, they use uncertainty as the level of input certainty captured by the model rather than irreducible environmental uncertainty.…”