Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut

Wang, Yangtao; Shen, Xi; Hu, Shell Xu; Yuan, Yuan; Crowley, James L.; Vaufreydaz, Dominique

doi:10.1109/cvpr52688.2022.01414

Cited by 69 publications

(38 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unlike the previous attempts which only consider foreground-background partition [36,38,40], or divide each image into a fixed number of clusters [26], we argue that it is important to consider different images distinguishably, due to the complexity of various scenarios. We thus propose the Dynamic Clustering Network (DCN) to cluster the pixels into dynamic semantic groups for each image.…”

Section: Introductionmentioning

confidence: 85%

“…For example, Simeoni et al [36] proposed a series of hand-made rules to chose pixels belonging to same object according to their feature similarity, achieving unsupervised object discovery and detection. Wang et al [40] introduce normalized cuts [33] on the affinity graph constructed by pixel embeddings from DINO to divide foreground and background in an image, for unsupervised object discovery and saliency detection task. For semantic segmentation task, Hamilton et al [12] train a segmentation head by distilling the feature correspondences, which further encourages pixel features to form compact clusters and learn better pixel-level representations.…”

Section: Self-supervised Vision Transformers and Applicationsmentioning

confidence: 99%

See 1 more Smart Citation

Dynamic Clustering Network for Unsupervised Semantic Segmentation

Li¹,

Wang²,

Cheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

Figure 1. t-SNE visualization of the pixel-level representations produced by self-supervised ViT and the corresponding cluster centers discovered by the proposed dynamic clustering network (DCN). It can be found that these pixel representations present different groups in the representation space, which are associated with the semantics of the pixels. We mark the underlying groups found by the DCN by different colors. Based on the pixel embeddings produced by self-supervised ViT, the proposed DCN successfully finds the underlying semantic groups dynamically for different images with scenes of different complexity.

show abstract

Section: Introductionmentioning

confidence: 85%

Section: Self-supervised Vision Transformers and Applicationsmentioning

confidence: 99%

Dynamic Clustering Network for Unsupervised Semantic Segmentation

Li¹,

Wang²,

Cheng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, vision transformers have been adapted to broader domains, including point cloud [63] and video understanding [2,61]. Another line of works focuses on transferring pretrained knowledge of vision transformers in an unsupervised or semi-supervised manner by leveraging self-distillation [8], semantics reallocation [21], seed propagation [36], and normalized cut [47].…”

Section: Related Workmentioning

confidence: 99%

“…First, we compare our PAVER model against competitive baselines for predicting 360 • video saliency based on optical flow [49], gradient flow [46], generative adversarial networks [33], and class activation map with optical flow [11]. We also report the performance of unsupervised saliency detection and unsupervised object discovery models based on vision transformers, including TS-CAM [21], DINO [8], LOST [36] and TokenCut [47]. For a fair comparison with our approach, we use the identical local patch projection module in Sec.…”

Section: Experiments Settingmentioning

confidence: 99%

“…CC AUC-J AUC-B Motion Magnitude [49] 0.288 0.687 0.642 ConsistentVideoSal [46] 0.085 0.547 0.532 SalGAN [33] 0.312 0.717 0.692 Equirectangular [11] 0.337 0.839 0.783 CubePad (Static) [11] 0.448 0.881 0.852 CubePad (CLSTM) [11] 0.420 0.898 0.859 TS-CAM [21] 0.414 0.831 0.802 DINO [8] 0.406 0.850 0.831 LOST [36] 0.444 0.809 0.786 TokenCut [47] 0…”

Section: Experiments Settingmentioning

confidence: 99%

See 1 more Smart Citation

Panoramic Vision Transformer for Saliency Detection in 360° Videos

Yun¹,

Lee²,

Kim³

2022

Preprint

View full text Add to dashboard Cite

360 • video saliency detection is one of the challenging benchmarks for 360 • video understanding since non-negligible distortion and discontinuity occur in the projection of any format of 360 • videos, and capture-worthy viewpoint in the omnidirectional sphere is ambiguous by nature. We present a new framework named Panoramic Vision Transformer (PAVER). We design the encoder using Vision Transformer with deformable convolution, which enables us not only to plug pretrained models from normal videos into our architecture without additional modules or finetuning but also to perform geometric approximation only once, unlike previous deep CNN-based approaches. Thanks to its powerful encoder, PAVER can learn the saliency from three simple relative relations among local patch features, outperforming state-of-the-art models for the Wild360 benchmark by large margins without supervision or auxiliary information like class activation. We demonstrate the utility of our saliency prediction model with the omnidirectional video quality assessment task in VQA-ODV, where we consistently improve performance without any form of supervision, including head movement.

show abstract