Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers

Ru, Lixiang; Zhan, Yibing; Yu, Baosheng; Du, Bo

doi:10.1109/cvpr52688.2022.01634

Cited by 98 publications

(53 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Observe that we improved BPM state-of-the-art by 3.91 mIou% points and BPM+CRF by 5.4 mIoU%, both w.r.t. AFA [39], owning so far the best accuracy on both. B.…”

Section: Comparisons With State-of-the-artmentioning

confidence: 99%

“…This last is connected with the class prediction, using a BCE prediction loss. More recently, Vision Transformers [15] are emerging as an alternative to generate CAM [58,39]. Our method is the first one using only ViT without CAM to generate baseline pseudo-masks.…”

Section: Related Workmentioning

confidence: 99%

“…Dino [5] downstream task segments foreground from background for single class images, differing from WSSS. Only recently ViT contributed to WSSS with MCTformer [58] and AFA [39], though both resort to CAM. MCTformer exploits ViT attention mechanism to obtain localization maps.…”

Section: Related Workmentioning

confidence: 99%

“…Our ViT-PCM, with backbone ViT-S/16, is green-dashed, and ViT-B/16 is green-continuous. We can observe that most of the shown methods are multi-stage (see also [3,39]), and boosting the BPM asks for a significant increase of parameters. Table 3 shows the accuracy between CAM and PCM on different backbones and the amount of parameters required.…”

Section: Comparisons With State-of-the-artmentioning

confidence: 99%

“…[6] and [41] use PSA; CONTA [14] and ReCAM [12] use IRNet while [30] using both. AFA [39] use PAC [44].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation

Rossetti¹,

Zappia²,

Sanzari³

et al. 2022

Preprint

View full text Add to dashboard Cite

Weakly Supervised Semantic Segmentation (WSSS) research has explored many directions to improve the typical pipeline CNN plus class activation maps (CAM) plus refinements, given the image-class label as the only supervision. Though the gap with the fully supervised methods is reduced, further abating the spread seems unlikely within this framework. On the other hand, WSSS methods based on Vision Transformers (ViT) have not yet explored valid alternatives to CAM. ViT features have been shown to retain a scene layout, and object boundaries in self-supervised learning. To confirm these findings, we prove that the advantages of transformers in self-supervised methods are further strengthened by Global Max Pooling (GMP), which can leverage patch features to negotiate pixel-label probability with class probability. This work proposes a new WSSS method dubbed ViT-PCM (ViT Patch-Class Mapping), not based on CAM. The end-to-end presented network learns with a single optimization process, refined shape and proper localization for segmentation masks. Our model outperforms the state-of-the-art on baseline pseudo-masks (BPM), where we achieve 69.3% mIoU on Pas-calVOC 2012 val set. We show that our approach has the least set of parameters, though obtaining higher accuracy than all other approaches. In a sentence, quantitative and qualitative results of our method reveal that ViT-PCM is an excellent alternative to CNN-CAM based architectures.

show abstract

“…Observe that we improved BPM state-of-the-art by 3.91 mIou% points and BPM+CRF by 5.4 mIoU%, both w.r.t. AFA [39], owning so far the best accuracy on both. B.…”

Section: Comparisons With State-of-the-artmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%