Discovering Causal Signals in Images

López-Paz, David; Nishihara, Robert; Chintala, Soumith; Schölkopf, Bernhard; Bottou, Léon

doi:10.1109/cvpr.2017.14

Cited by 171 publications

(133 citation statements)

References 14 publications

(23 reference statements)

Supporting

Mentioning

132

Contrasting

Order By: Relevance

“…In our experiments with people's faces, eyes and lips are considered to be the direct proxy for gender attractiveness, and nose regions for being in a certain age group. As a potential future direction, we plan to further analyze the interpretability in fairness using causal reasoning [31].…”

Section: Discussionmentioning

confidence: 99%

Discovering Fair Representations in the Data Domain

Quadrianto

Sharmanska

Thomas

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Interpretability and fairness are critical in computer vision and machine learning applications, in particular when dealing with human outcomes, e.g. inviting or not inviting for a job interview based on application materials that may include photographs. One promising direction to achieve fairness is by learning data representations that remove the semantics of protected characteristics, and are therefore able to mitigate unfair outcomes. All available models however learn latent embeddings which comes at the cost of being uninterpretable. We propose to cast this problem as data-to-data translation, i.e. learning a mapping from an input domain to a fair target domain, where a fairness definition is being enforced. Here the data domain can be images, or any tabular data representation. This task would be straightforward if we had fair target data available, but this is not the case. To overcome this, we learn a highly unconstrained mapping by exploiting statistics of residuals -the difference between input data and its translated version -and the protected characteristics. When applied to the CelebA dataset of face images with gender attribute as the protected characteristic, our model enforces equality of opportunity by adjusting the eyes and lips regions. Intriguingly, on the same dataset we arrive at similar conclusions when using semantic attribute representations of images for translation. On face images of the recent DiF dataset, with the same gender attribute, our method adjusts nose regions. In the Adult income dataset, also with protected gender attribute, our model achieves equality of opportunity by, among others, obfuscating the wife and husband relationship. Analyzing those systematic changes will allow us to scrutinize the interplay of fairness criterion, chosen protected characteristics, and prediction performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Discovering Fair Representations in the Data Domain

Quadrianto

Sharmanska

Thomas

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…Recently, the deep learning community started to tackle complex visual reasoning problems such as relationship detection [26], object recognition [9], abstract reasoning [33] or visual causality [25], while more theoretical work attempt to formalize relational reasoning [7].…”

Section: Related Work and Contributionsmentioning

confidence: 99%

MUREL: Multimodal Relational Reasoning for Visual Question Answering

Cadène¹,

Ben-younes

Cord

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

280

166

View full text Add to dashboard Cite

Multimodal attentional networks are currently state-ofthe-art models for Visual Question Answering (VQA) tasks involving real images. Although attention allows to focus on the visual content relevant to the question, this simple mechanism is arguably insufficient to model complex reasoning features required for VQA or other high-level tasks.In this paper, we propose MuRel, a multimodal relational network which is learned end-to-end to reason over real images. Our first contribution is the introduction of the MuRel cell, an atomic reasoning primitive representing interactions between question and image regions by a rich vectorial representation, and modeling region relations with pairwise combinations. Secondly, we incorporate the cell into a full MuRel network, which progressively refines visual and question interactions, and can be leveraged to define visualization schemes finer than mere attention maps.We validate the relevance of our approach with various ablation studies, and show its superiority to attentionbased methods on three datasets: VQA 2.0, VQA-CP v2 and TDIUC. Our final MuRel network is competitive to or outperforms state-of-the-art results in this challenging context.Our code is available: github.com/Cadene/ murel.bootstrap.pytorch * Equal contribution DonutStep #1Step #2 Step #3 MuRel network ✔

show abstract

“…It is of increasing research interests in computer vision that attempts to borrow useful analysis tools from causality. Typical works are concerning object tracking [21,52], interpretable Learning [15], image classification [7,28], and image generation [5,16]. More recently, Wang et al [49] proposes the VC R-CNN framework for visual representation learning.…”

Section: Causality In Vision and Languagementioning

confidence: 99%

DeVLBert

Zhang

Jiang²,

Wang

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

In this paper, we propose to investigate the problem of out-ofdomain visio-linguistic pretraining, where the pretraining data distribution differs from that of downstream data on which the pretrained model will be fine-tuned. Existing methods for this problem are purely likelihood-based, leading to the spurious correlations and hurt the generalization ability when transferred to out-of-domain downstream tasks. By spurious correlation, we mean that the conditional probability of one token (object or word) given another one can be high (due to the dataset biases) without robust (causal) relationships between them. To mitigate such dataset biases, we propose a Deconfounded Visio-Linguistic Bert framework, abbreviated as DeVLBert, to perform intervention-based learning. We borrow the idea of the backdoor adjustment from the research field of causality and propose several neural-network based architectures for Bert-style out-of-domain pretraining. The quantitative results on three downstream tasks, Image Retrieval (IR), Zero-shot IR, and Visual Question Answering, show the effectiveness of DeVLBert by boosting generalization ability. CCS CONCEPTS • Computing methodologies → Transfer learning.

show abstract

Discovering Causal Signals in Images

Cited by 171 publications

References 14 publications

Discovering Fair Representations in the Data Domain

Discovering Fair Representations in the Data Domain

MUREL: Multimodal Relational Reasoning for Visual Question Answering

DeVLBert

Contact Info

Product

Resources

About