What Are People Doing About XAI User Experience? A Survey on AI Explainability Research and Practice

Ferreira, Juliana Jansen; Monteiro, Mateus de Souza

doi:10.1007/978-3-030-49760-6_4

Cited by 44 publications

(34 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…More generally, all the metrics discussed in this paper should be tested against a user experiment. As shown by [6,8], there is a great variety of possible experimental setups depending on what should be explained, in which context, and for who the explanation is targeted.…”

Section: Discussionmentioning

confidence: 99%

Metrics for saliency map evaluation of deep learning explanation methods

Gomez¹,

Fréour²,

Mouchère³

2022

Preprint

View full text Add to dashboard Cite

Due to the black-box nature of deep learning models, there is a recent development of solutions for visual explanations of CNNs. Given the high cost of user studies, metrics are necessary to compare and evaluate these different methods. In this paper, we critically analyze the Deletion Area Under Curve (DAUC) and Insertion Area Under Curve (IAUC) metrics proposed by Petsiuk et al. (2018). These metrics were designed to evaluate the faithfulness of saliency maps generated by generic methods such as Grad-CAM or RISE. First, we show that the actual saliency score values given by the saliency map are ignored as only the ranking of the scores is taken into account. This shows that these metrics are insufficient by themselves, as the visual appearance of a saliency map can change significantly without the ranking of the scores being modified. Secondly, we argue that during the computation of DAUC and IAUC, the model is presented with images that are out of the training distribution which might lead to an unreliable behavior of the model being explained. To complement DAUC/IAUC, we propose new metrics that quantify the sparsity and the calibration of explanation methods, two previously unstudied properties. Finally, we give general remarks about the metrics studied in this paper and discuss how to evaluate them in a user study.

show abstract

Section: Discussionmentioning

confidence: 99%

Metrics for saliency map evaluation of deep learning explanation methods

Gomez¹,

Fréour²,

Mouchère³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…While several prior literature surveys have sought to collate and organize a list of interpretability needs, our framework makes some key advances to provide a more nuanced understanding these needs. First, where prior surveys focus primarily on computer science subdisciplines [44,57,87], our framework incorporates these insights and extends them by looking to the legal literature [31,55,118,132] and research on participatory action and design [50,108]. As a result, our framework is able to surface objectives such as "contesting a decision" (O7) or "understanding how one's data is being used" (O5) that prior surveys did not identify.…”

Section: Distilling Stakeholder Needs Into Goals Objectives and Tasksmentioning

confidence: 99%

“…Part of this disconnect stems from the difficulty in identifying and characterizing different stakeholders and their interpretability needs. A growing body of work has engaged with this problem, proposing an ecosystem of stakeholders [96,113], and conducting literature surveys [44,57,87,131] and interview studies [27,58,114] to understand their goals. Resultant frameworks typically adopt one of two approaches: they either categorize stakeholders by their expertise (using labels such as "experts", "novices", or "nonexperts" [57,87,131]) or by their functional role in the ecosystem (e.g., "executives" and "engineers" [17], model "breakers" and "consumers" [58], or model "operators" and "executors" [113]).…”

Section: Introductionmentioning

confidence: 99%

Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs

Suresh

Gomez

Nam

et al. 2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

To ensure accountability and mitigate harm, it is critical that diverse stakeholders can interrogate black-box automated systems and find information that is understandable, relevant, and useful to them. In this paper, we eschew prior expertise-and role-based categorizations of interpretability stakeholders in favor of a more granular framework that decouples stakeholders' knowledge from their interpretability needs. We characterize stakeholders by their formal, instrumental, and personal knowledge and how it manifests in the contexts of machine learning, the data domain, and the general milieu. We additionally distill a hierarchical typology of stakeholder needs that distinguishes higher-level domain goals from lower-level interpretability tasks. In assessing the descriptive, evaluative, and generative powers of our framework, we find our more nuanced treatment of stakeholders reveals gaps and opportunities in the interpretability literature, adds precision to the design and comparison of user studies, and facilitates a more reflexive approach to conducting this research.

show abstract

“…For instance, there are works on developing algorithms and novel DL architectures in XAI to add explainability to the models [ 42 , 43 , 44 , 45 , 46 ]. In comparison, there is also work that considers user experience and user requirements for XAI [ 7 , 8 , 9 , 10 , 47 ], and evaluates algorithms and models with user studies [ 48 ]. However, analyzing and categorizing XAI algorithms is not the focus of this paper.…”

Section: Classifying Hcml Researchmentioning

confidence: 99%

“…Some of the most common non-expert-user issues are explainability [ 6 , 7 , 8 , 9 , 10 ], interpretability [ 11 , 12 , 13 , 14 ], privacy and security [ 15 , 16 , 17 , 18 , 19 , 20 ], reliability [ 21 , 22 , 23 , 24 , 25 ], and fairness [ 26 , 27 , 28 , 29 , 30 , 31 ]. These categories emerged as contributing research areas of HCML, and each plays a role in the broader goal of improving the usability and adoptability of AI systems.…”

Section: Introductionmentioning

confidence: 99%

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

Kaluarachchi

Reis

Nanayakkara

2021

Sensors

View full text Add to dashboard Cite

After Deep Learning (DL) regained popularity recently, the Artificial Intelligence (AI) or Machine Learning (ML) field is undergoing rapid growth concerning research and real-world application development. Deep Learning has generated complexities in algorithms, and researchers and users have raised concerns regarding the usability and adoptability of Deep Learning systems. These concerns, coupled with the increasing human-AI interactions, have created the emerging field that is Human-Centered Machine Learning (HCML). We present this review paper as an overview and analysis of existing work in HCML related to DL. Firstly, we collaborated with field domain experts to develop a working definition for HCML. Secondly, through a systematic literature review, we analyze and classify 162 publications that fall within HCML. Our classification is based on aspects including contribution type, application area, and focused human categories. Finally, we analyze the topology of the HCML landscape by identifying research gaps, highlighting conflicting interpretations, addressing current challenges, and presenting future HCML research opportunities.

show abstract

What Are People Doing About XAI User Experience? A Survey on AI Explainability Research and Practice

Cited by 44 publications

References 69 publications

Metrics for saliency map evaluation of deep learning explanation methods

Metrics for saliency map evaluation of deep learning explanation methods

Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

Contact Info

Product

Resources

About