Sanity Checks for Saliency Metrics

Tomsett, Richard; Harborne, Daniel; Chakraborty, Supriyo; Gurram, Prudhvi; Preece, Alun

doi:10.1609/aaai.v34i04.6064

Cited by 90 publications

(81 citation statements)

References 13 publications

(27 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In principle this could be used to improve the interactive feedback loop proposed for the ( P2 ) scenario. Another direction for future work is to complement the metrics of LEAF framework with a set of sanity checks based on measures of statistical reliability, as suggested in Tomsett et al (2020) for saliency metrics.…”

Section: Discussionmentioning

confidence: 99%

To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods

Amparore

Perotti

Bajardi

2021

PeerJ Computer Science

View full text Add to dashboard Cite

The main objective of eXplainable Artificial Intelligence (XAI) is to provide effective explanations for black-box classifiers. The existing literature lists many desirable properties for explanations to be useful, but there is a scarce consensus on how to quantitatively evaluate explanations in practice. Moreover, explanations are typically used only to inspect black-box models, and the proactive use of explanations as a decision support is generally overlooked. Among the many approaches to XAI, a widely adopted paradigm is Local Linear Explanations—with LIME and SHAP emerging as state-of-the-art methods. We show that these methods are plagued by many defects including unstable explanations, divergence of actual implementations from the promised theoretical properties, and explanations for the wrong label. This highlights the need to have standard and unbiased evaluation procedures for Local Linear Explanations in the XAI field. In this paper we address the problem of identifying a clear and unambiguous set of metrics for the evaluation of Local Linear Explanations. This set includes both existing and novel metrics defined specifically for this class of explanations. All metrics have been included in an open Python framework, named LEAF. The purpose of LEAF is to provide a reference for end users to evaluate explanations in a standardised and unbiased way, and to guide researchers towards developing improved explainable techniques.

show abstract

Section: Discussionmentioning

confidence: 99%

To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods

Amparore

Perotti

Bajardi

2021

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…OPEN ACCESS validate, 41 with some failing basic sanity checks. 42 This would preclude the use of neural network models for high-stakes decision support.…”

Section: Llmentioning

confidence: 99%

Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI

Tomsett¹,

Preece

Braines³

et al. 2020

Patterns

Self Cite

View full text Add to dashboard Cite

Artificial intelligence (AI) systems hold great promise as decision-support tools, but we must be able to identify and understand their inevitable mistakes if they are to fulfill this potential. This is particularly true in domains where the decisions are high-stakes, such as law, medicine, and the military. In this Perspective, we describe the particular challenges for AI decision support posed in military coalition operations. These include having to deal with limited, low-quality data, which inevitably compromises AI performance. We suggest that these problems can be mitigated by taking steps that allow rapid trust calibration so that decision makers understand the AI system's limitations and likely failures and can calibrate their trust in its outputs appropriately. We propose that AI services can achieve this by being both interpretable and uncertainty-aware. Creating such AI systems poses various technical and human factors challenges. We review these challenges and recommend directions for future research.

show abstract

“…Since the proper interpretation method is to rank the features most sensitive to the model's decision, it seems natural to consider the Spearman rank correlation [58] to compare the similarity between explanations. Prior work has provided theoretical and experimental arguments in line with this choice [50,49,59,46].…”

Section: Efficiencymentioning

confidence: 99%

Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis

Fel¹,

Cadène²,

Chalvidal³

et al. 2021

Preprint

View full text Add to dashboard Cite

We describe a novel attribution method which is grounded in Sensitivity Analysis and uses Sobol indices. Beyond modeling the individual contributions of image regions, Sobol indices provide an efficient way to capture higher-order interactions between image regions and their contributions to a neural network's prediction through the lens of variance. We describe an approach that makes the computation of these indices efficient for high-dimensional problems by using perturbation masks coupled with efficient estimators to handle the high dimensionality of images. Importantly, we show that the proposed method leads to favorable scores on standard benchmarks for vision (and language models) while drastically reducing the computing time compared to other black-box methods -even surpassing the accuracy of state-of-the-art white-box methods which require access to internal representations. Our code is freely available: github.com/fel-thomas/ Sobol-Attribution-Method.* Equal contribution † Work done before April 2021 and joining Tesla Preprint. Under review.

show abstract

Sanity Checks for Saliency Metrics

Cited by 90 publications

References 13 publications

To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods

To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods

Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI

Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis

Contact Info

Product

Resources

About