Interpretability is in the Eye of the Beholder: Human Versus Artificial Classification of Image Segments Generated by Humans Versus XAI

Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type

“…Thus, our tasks do not easily scale to more complex categorization demands. Some of these implementation issues could be solved by minor changes to the procedure, such as performing verbal instead of manual labelling (e.g., [56]) or presenting a category label beforehand and then having participants indicate whether it matches the image or not (e.g., [67,68]).…”

“…[45]). To avoid the pitfalls of inferring the suitability of CNN attention maps from their similarity to humans, external criteria are needed, such as whether they are interpretable and support human task performance [68,77,78].…”

How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps

“…Several studies reported human performance differences depending on attribution methods. These effects were most prominent when participants had to classify image segments that were generated by different attribution methods (Biessmann & Refiano, 2019;John et al, 2021;Lu et al, 2021;Müller, Thoß, et al, 2024;Selvaraju et al, 2017). Obviously, this task depends on the attributions as it requires a classification of highly restricted information.…”

“…Participants strongly benefitted from one method of generating saliency maps, whereas two other methods even impaired performance as they either highlighted too few or too many positions. Moreover, the superiority of attribution methods can depend on other factors such as image type (Müller, Thoß, et al, 2024). When the class was defined by singular objects, participants could more easily classify segments generated by XRAI than Grad-CAM, while the reverse was true for complex scenes.…”

“…Their results typically diverge from those obtained in user studies that require humans to actually work with saliency maps. This is the case both for metrics that assess an XAI method's fidelity to the AI model (Biessmann & Refiano, 2019;Colin et al, 2022;Lu et al, 2021;Müller, Thoß, et al, 2024) and for those that assess its similarity to human-selected image areas (Kim et al, 2022;Nguyen et al, 2021). Accordingly, user studies seem inevitable in the evaluation of XAI, if the aim is to generate explanations that can support human decision making.…”