Anti-Adversarially Manipulated Attributions for Weakly Supervised Semantic Segmentation and Object Localization

Abstract

Obtaining accurate pixel-level localization from class labels is a crucial process in weakly supervised semantic segmentation and object localization. Attribution maps from a trained classifier are widely used to provide pixel-level localization, but their focus tends to be restricted to a small discriminative region of the target object. An AdvCAM is an attribution map of an image that is manipulated to increase the classification score produced by a classifier before the final softmax or sigmoid layer. This manipulation is realized in an anti-adversarial manner, so that the original image is perturbed along pixel gradients in directions opposite to those used in an adversarial attack. This process enhances non-discriminative yet class-relevant features, which make an insufficient contribution to previous attribution maps, so that the resulting AdvCAM identifies more regions of the target object. In addition, we introduce a new regularization procedure that inhibits the incorrect attribution of regions unrelated to the target object and the excessive concentration of attributions on a small region of the target object. Our method achieves a new state-of-the-art performance in weakly and semi-supervised semantic segmentation, on both the PASCAL VOC 2012 and MS COCO 2014 datasets. In weakly supervised object localization, it achieves a new state-of-the-art performance on the CUB-200-2011 and ImageNet-1K datasets.

Weighted and Mumford-Shah losses for weakly supervised semantic segmentation

“…Kim et al [4] proposed a discriminative region suppression module and a localization map optimization strategy. Lee et al [5] employed adversarial learning to introduce perturbations on image pixel gradients to explore more class-relevant features. Furthermore, self-supervised contrastive learning methods have been widely adopted.…”

3D Visual Discomfort Assessment With a Weakly Supervised Graph Convolution Neural Network Based on Inaccurately Labeled EEG

“…Therefore, weakly supervised learning has recently gained attention because it allows accurate classification models to be constructed using low-quality or limited labeled data [36]. Weakly supervised learning is widely used in tasks such as semantic segmentation and object detection, where weakly annotated data, including image-level category labels, are used to train models that can generate pixel-level predictions [37], [38]. Three types of weakly supervised learning strategies are available: incomplete supervision, inexact supervision, and inaccurate supervision [36].…”

Spatial-Aware Token for Weakly Supervised Object Localization

“…Weakly supervised object localization (WSOL) aims to localize objects with only image-level labels available. Since no expensive bounding box or pixel-level annotations are required, WSOL significantly reduces the cost of manual annotations and has attracted increasing attention in the research community [22,37,34,12,14,4,36]. As a representative work, CAM [41] extracts class activation maps from the classifier as localization maps.…”

“…However, CAM is usually coarse and focuses on the most discriminative regions, leading to imprecise and incomplete localization results. To solve these problems, many CNNbased methods have been proposed, such as adversarial erasure [23,38,6,16], divergent activation [33,35,23], seed region growing [27,39], regularization [19,14,40], feature refining [1,34,27], regression-based [32,36,9].…”