PsyNet: Self-Supervised Approach to Object Localization Using Point Symmetric Transformation

Baek, Kyungjune; Lee, Minhyun; Shim, Hyunjung

doi:10.1609/aaai.v34i07.6615

Cited by 35 publications

(44 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other methods use technique such as CAM or analogous techniques [15,19,28,81,85,111] as a form of weakly supervised saliency or localisation maps. Recent works have suggested that saliency methods can also be applied to self-supervised networks [13,35], e.g. for object colocalisation [13].…”

Section: Related Workmentioning

confidence: 99%

Self-supervised object detection from audio-visual correspondence

Afouras¹,

Asano²,

Fagan³

et al. 2021

Preprint

View full text Add to dashboard Cite

We tackle the problem of learning object detectors without supervision. Differently from weakly-supervised object detection, we do not assume image-level class labels. Instead, we extract a supervisory signal from audio-visual data, using the audio component to "teach" the object detector. While this problem is related to sound source localisation, it is considerably harder because the detector must classify the objects by type, enumerate each instance of the object, and do so even when the object is silent. We tackle this problem by first designing a self-supervised framework with a contrastive objective that jointly learns to classify and localise objects. Then, without using any supervision, we simply use these self-supervised labels and boxes to train an image-based object detector. With this, we outperform previous unsupervised and weakly-supervised detectors for the task of object detection and sound source localization. We also show that we can align this detector to ground-truth classes with as little as one label per pseudo-class, and show how our method can learn to detect generic objects that go beyond instruments, such as airplanes and cats.

show abstract

Section: Related Workmentioning

confidence: 99%

Self-supervised object detection from audio-visual correspondence

Afouras¹,

Asano²,

Fagan³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Differently, semantic information of objects is agnostic in self-supervised learning. Baek et al [23] proposed to utilize point symmetric transformation as the self-supervision signal to promote class-agnostic heatmap extraction in the object localization task. The aforementioned methods just utilize visual information to mine effective supervision.…”

Section: Object Localizationmentioning

confidence: 99%

Class-aware Sounding Objects Localization via Audiovisual Correspondence

Hu¹,

Wei²,

Qu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Audiovisual scenes are pervasive in our daily life. It is commonplace for humans to discriminatively localize different sounding objects but quite challenging for machines to achieve class-aware sounding objects localization without category annotations, i.e., localizing the sounding object and recognizing its category. To address this problem, we propose a two-stage step-by-step learning framework to localize and recognize sounding objects in complex audiovisual scenarios using only the correspondence between audio and vision. First, we propose to determine the sounding area via coarse-grained audiovisual correspondence in the single source cases. Then visual features in the sounding area are leveraged as candidate object representations to establish a category-representation object dictionary for expressive visual character extraction. We generate class-aware object localization maps in cocktail-party scenarios and use audiovisual correspondence to suppress silent areas by referring to this dictionary. Finally, we employ category-level audiovisual consistency as the supervision to achieve fine-grained audio and sounding object distribution alignment. Experiments on both realistic and synthesized videos show that our model is superior in localizing and recognizing objects as well as filtering out silent ones. We also transfer the learned audiovisual network into the unsupervised object detection task, obtaining reasonable performance.

show abstract

“…Grad-CAM [31] and Grad-CAM++ [3] generalize CAM [56], so that CAMs can be obtained in any CNN-based classification models. The CAM technique derived from the classification network has been widely used for other weakly supervised visual tasks, such as localization [2,48], detection [39,43,55], segmentation [1,23,44].…”

Section: Related Work 21 Class Activation Mapmentioning

confidence: 99%

“…It is easily obtained by computing a weighted sum of the feature maps of the last convolutional layer. In fact, we can also obtain a class-agnostic activation map (CAAM [2]) by computing the sum of the feature maps directly, which indicates the spatial distribution of the embedded features. To describe our motivation visually, in Figure 1 we show some validation images misclassified by a pre-trained ResNet-50 [14] model with cross entropy loss on ImageNet [29].…”

Section: Introductionmentioning

confidence: 99%

CAM-loss: Towards Learning Spatially Discriminative Feature Representations

Wang¹,

Xiao²,

Han³

et al. 2021

Preprint

View full text Add to dashboard Cite

The backbone of traditional CNN classifier is generally considered as a feature extractor, followed by a linear layer which performs the classification. We propose a novel loss function, termed as CAM-loss, to constrain the embedded feature maps with the class activation maps (CAMs) which indicate the spatially discriminative regions of an image for particular categories. CAM-loss drives the backbone to express the features of target category and suppress the features of non-target categories or background, so as to obtain more discriminative feature representations. It can be simply applied in any CNN architecture with neglectable additional parameters and calculations. Experimental results show that CAM-loss is applicable to a variety of network structures and can be combined with mainstream regularization methods to improve the performance of image classification. The strong generalization ability of CAMloss is validated in the transfer learning and few shot learning tasks. Based on CAM-loss, we also propose a novel CAAM-CAM matching knowledge distillation method. This method directly uses the CAM generated by the teacher network to supervise the CAAM generated by the student network, which effectively improves the accuracy and convergence rate of the student network.

show abstract

PsyNet: Self-Supervised Approach to Object Localization Using Point Symmetric Transformation

Cited by 35 publications

References 21 publications

Self-supervised object detection from audio-visual correspondence

Self-supervised object detection from audio-visual correspondence

Class-aware Sounding Objects Localization via Audiovisual Correspondence

CAM-loss: Towards Learning Spatially Discriminative Feature Representations

Contact Info

Product

Resources

About