Comparing User Perception of Explanations Developed with XAI Methods

Abstract

published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User

Assessing Perceived Trust and Satisfaction with Multiple Explanation Techniques in XAI-Enhanced Learning Analytics

“…Observing the positively stated TIA items, the global model performed higher on the measured trust level with items Q7, Q9, and Q10, related to the security of the explanation, its dependency, and its reliability. These results are similar to the findings of [41], where global SHAP explanations were evaluated with a slightly higher perceived trust on a simple one-scale measure compared to local SHAP explanations. The trust evaluation method utilized in our study allowed us to evaluate the initial user trust in a more detailed view.…”

“…Furthermore, we confirm the findings of [9], who reported that trust has been found to increase when the reasoning for the AI system's decision is provided (explanation techniques A, B, C, and G) and to decrease when information on sources of uncertainty is shared with the user (explanation techniques D, E, and F scored comparatively lower on the TIA questionnaire with both groups of students). Comparison of the ESS results in the group of master's students, who were presented with global and local explanations, also confirms the findings of [41] in our study setting, where higher user satisfaction was reported with global explanation G compared to local feature explanations (A-C).…”

“…The study reported the lowest user satisfaction with causal explanations and the highest levels of trust with intentional explanations, while mixed explanations led to the best functional understanding of the system. Related evaluation of understandability, usefulness, trustworthiness, informativeness, and satisfaction with explanations, generated with popular XAI methods (LIME [38], SHAP [39], and Partial Dependence Plots or PDP [40]) was conducted by [41], reporting higher satisfaction with global explanations with novice users compared to local feature explanations. Comparing the popular methods, PDP performed best on all evaluated criteria.…”

Towards Explainable Linguistic Summaries

“…For this, in-depth user studies should be carried out. It is important to ask the end user how they perceived the explanations in terms of various aspects like understandability, usefulness, trustworthiness, and helpfulness [53], [54]. Moreover, it is also necessary to objectively test whether these explanations really have been understood and can be utilized efficiently.…”

Personal Characteristics and Explanations in Technology-Enhanced Environments: An Exploratory Study in Computer Science Education

“…However, we can hypothesize that this explanation was consulted very little and therefore had little to no influence on the students, as during the focus group, none of the students who received explanations recalled this global explanation. However, further work is needed to investigate this topic in more detail and thus complement work that has compared users' perceptions of different types of explanations, and which has not validated the hypothesis that AI novices prefer local explanations to global explanations (Aechtner et al, 2022).…”