Diffusion Models for Counterfactual Explanations

Jeanneret, Guillaume; Simon, Loïc; Jurie, Frédéric

doi:10.48550/arxiv.2203.15636

Cited by 4 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, similar approaches have been used to guide Diffusion Models 3 [44], in which the diffusion procedure is conditioned on the basis of the classifier gradient. Such methods have shown promising results in providing post-hoc explanations of classification models [45].…”

Section: Activation Maximisation (Am)mentioning

confidence: 99%

“…In this work, we decided to combine the feature visualizations produced with the AM method with Stable Diffusion (SD-AM). Despite some work on the generation of counter-factual explanations [45,105], the investigation of diffusion models trained on large-scale data, such as Stable Diffusion, as tools that allow a better understanding of classification models is a novel approach. In Figure 10, we present results of applying SD-AM to obtain hypericons related to the power class.…”

Section: Denoising With Stable Diffusionmentioning

confidence: 99%

See 1 more Smart Citation

Hypericons for Abstract Concepts: Non-traditional Approaches to Visual Explainability

Pandiani

Lazzari

Erp

et al. 2023

Preprint

View full text Add to dashboard Cite

In the cultural heritage (CH) field, abstract concepts–such as comfort, power, or freedom–are considered powerful tools to index cultural visual data. Simultaneously, the ever-increasing number of images is too great to index manually, so computer vision (CV)-based methods are employed to automatically classify images. The CV field, however, has paid scarce attention to image classification based on abstract concepts, partly because it has historically favored tasks and datasets for purportedly objective, concrete categories. More subjective and culturally-coded classes, like abstract concepts, more explicitly require interpretability of predictions by automatic models, given the potential to echo harmful bias. This is especially true for their detection in CH collections, given that they tend to contain many `ethically sensitive' depictions. To address the gap in CV-based interpretable methods for automatic classification of abstract concepts, we (1) present ARTstract, a dataset of cultural images and their evocation of certain abstract concepts (2) report baseline model performances on ARTstract for the task of image classification by abstract concepts, and, critically, (3) use them as a case study of traditional and non-traditional approaches to visual interpretability, inspired by [Offert \& Bell 2021]’s work. We introduce a unique approach to testing the interpretability of predictions by automatic models, using a combination of attribution maps (AM) and stable diffusion (SD). This approach has not been attempted before, and we experiment with it as a way to extend hermeneutic work back into the technical system that enables, unveiling its potential as a provider of lessons for developing systems that are interpretable-by-design.

show abstract

Section: Activation Maximisation (Am)mentioning

confidence: 99%

Section: Denoising With Stable Diffusionmentioning

confidence: 99%

Hypericons for Abstract Concepts: Non-traditional Approaches to Visual Explainability

Pandiani

Lazzari

Erp

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

ABC-GAN: Spatially Constrained Counterfactual Generation for Image Classification Explanations

Mindlin,

Schilling,

Cimiano

2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

Hypericons for interpretability: decoding abstract concepts in visual data

Martinez Pandiani,

Lazzari,

Erp

et al. 2023

Int J Digit Humanities

View full text Add to dashboard Cite

In an era of information abundance and visual saturation, the need for resources to organise and access the vast expanse of visual data is paramount. Abstract concepts-such as comfort, power, or freedom-emerge as potent instruments to index and manage visual data, particularly in contexts like Cultural Heritage (CH). However, the variance and disparity in the visual signals that evoke a single abstract concept challenge conventional approaches to automatic visual management rooted in the Computer Vision (CV) field. This paper critically engages with the prevalent trend of automating high-level visual reasoning while placing exclusive reliance on visual signals, prominently featuring Convolutional Neural Networks (CNNs). We delve into this trend, scrutinising the knowledge sought by CNNs and the knowledge they ultimately encapsulate. In this endeavour, we accomplish three main objectives: (1) introduction of ARTstract, an extensive dataset encompassing cultural images that evoke specific abstract concepts; (2) presentation of baseline model performances on ARTstract to elucidate the intricate nuances of image classification based on abstract concepts; and, critically, (3) utilization of ARTstract as a case study to explore both traditional and non-traditional avenues of visual interpretability, a trajectory inspired by Offert and Bell (2021). To more comprehensively understand how CNNs assimilate and reflect cultural meanings, and to discern the echoes reverberating within these visions, we unveil SD-AM, a novel approach to explainability that collapses visuals into hypericon images through a fusion of feature visualization techniques and Stable Diffusion denoising. Overall, this study critically addresses abstract concept image classification’s challenges within the CNN paradigm. By embracing innovative methodologies and providing comprehensive analyses of explainability techniques, we make a substantial contribution to the broader discourse surrounding automatic high-level visual understanding, its interpretability, and the ensuing implications for comprehending culture within the digital era. Through our exploration, we illuminate the multifaceted trends, complexities, and opportunities that underlie the fusion of high-level visual reasoning and computer vision.

show abstract

Diffusion Models for Counterfactual Explanations

Cited by 4 publications

References 0 publications

Hypericons for Abstract Concepts: Non-traditional Approaches to Visual Explainability

Hypericons for Abstract Concepts: Non-traditional Approaches to Visual Explainability

ABC-GAN: Spatially Constrained Counterfactual Generation for Image Classification Explanations

Hypericons for interpretability: decoding abstract concepts in visual data

Contact Info

Product

Resources

About