Remember What You have drawn: Semantic Image Manipulation with Memory

Shi, Xiangxi; Wu, Zhonghua; Lin, Guosheng; Cai, Jianfei; Joty, Shafiq

doi:10.48550/arxiv.2107.12579

Cited by 3 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Semantic segmentation [29,35,15,14] is a task of classifying each pixel in an image into a specified category and has been applied in various fields [30,24]. State-of-the-art segmentation methods are usually based on the Fully Convolutional Network (FCN) [18], which uses a classification network as the backbone and replaces fully connected layers with convolutional layers to predict the dense segmentation map.…”

Section: Semantic Segmentationmentioning

confidence: 99%

Learning Meta-class Memory for Few-Shot Semantic Segmentation

Shi

Lin

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Currently, the state-of-the-art methods treat few-shot semantic segmentation task as a conditional foregroundbackground segmentation problem, assuming each class is independent. In this paper, we introduce the concept of meta-class, which is the meta information (e.g. certain middle-level features) shareable among all classes. To explicitly learn meta-class representations in few-shot segmentation task, we propose a novel Meta-class Memory based few-shot segmentation method (MM-Net), where we introduce a set of learnable memory embeddings to memorize the meta-class information during the base class training and transfer to novel classes during the inference stage. Moreover, for the k-shot scenario, we propose a novel image quality measurement module to select images from the set of support images. A high-quality class prototype could be obtained with the weighted sum of support image features based on the quality measure. Experiments on both PASCAL-5 i and COCO dataset shows that our proposed method is able to achieve state-of-the-art results in both 1shot and 5-shot settings. Particularly, our proposed MM-Net achieves 37.5% mIoU on the COCO dataset in 1-shot setting, which is 5.1% higher than the previous state-of-theart.

show abstract

Section: Semantic Segmentationmentioning

confidence: 99%

Learning Meta-class Memory for Few-Shot Semantic Segmentation

Shi

Lin

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Currently, state-of-the-art methods handle image semantic segmentation as a dense prediction task and adopt fully convolutional networks to make predictions [26], [27]. To make pixel-level dense predictions, encoder-decoder structures [28], [29], [30], [31], [17], [32], [33], [34], [35], [36], [37], [38], [39] are widely used to reconstruct high-resolution prediction maps. Typically an encoder gradually downsamples the feature maps, aiming to acquire large field-of-view and capture the semantic object information.…”

Section: A Semantic Segmentationmentioning

confidence: 99%

Cross-Image Region Mining with Region Prototypical Network for Weakly Supervised Segmentation

Liu

Kong²,

Hung³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Weakly supervised image segmentation trained with image-level labels usually suffers from inaccurate coverage of object areas during the generation of the pseudo groundtruth. This is because the object activation maps are trained with the classification objective and lack the ability to generalize. To improve the generality of the objective activation maps, we propose a region prototypical network (RPNet) to explore the cross-image object diversity of the training set. Similar object parts across images are identified via region feature comparison. Object confidence is propagated between regions to discover new object areas while background regions are suppressed. Experiments show that the proposed method generates more complete and accurate pseudo object masks, while achieving state-of-the-art performance on PASCAL VOC 2012 and MS COCO. In addition, we investigate the robustness of the proposed method on reduced training sets.

show abstract

“…Semantic segmentation [58][59][60][61][62] is a task of classifying each pixel in an image into a specified category and has been applied in various fields [63][64][65]. State-of-theart segmentation methods are usually based on the Fully Convolutional Network 2.3.…”

Section: Image Segmentation With Limited Supervision 231 Semantic Seg...mentioning

confidence: 99%

2D and 3D visual understanding with limited supervision

Wu¹

View full text Add to dashboard Cite

under various perturbations to effectively regularize unlabeled 3D points. In particular, we propose a novel Dual Adaptive Transformations (DAT) model for weakly supervised point cloud segmentation, where the dual adaptive transformations are performed via an adversarial strategy at both point-level and region-level, aiming at enforcing the local and structural smoothness constraints on 3D point clouds. Secondly, in Chapter 6, we observe that the straightforward way of applying consistency constraints to weakly supervised point cloud segmentation has two major limitations: unsatisfied pseudo labels due to the conventional confidence-based selection and insufficient consistency constraints due to discarding unreliable pseudo labels. Therefore, we propose a novel Reliability-Adaptive Consistency Network (RAC-Net) to use both prediction confidence and model uncertainty to measure the reliability of pseudo labels and apply consistency training on all unlabeled points while with different consistency constraints for different points based on the reliability of corresponding pseudo labels.

show abstract

Remember What You have drawn: Semantic Image Manipulation with Memory

Cited by 3 publications

References 21 publications

Learning Meta-class Memory for Few-Shot Semantic Segmentation

Learning Meta-class Memory for Few-Shot Semantic Segmentation

Cross-Image Region Mining with Region Prototypical Network for Weakly Supervised Segmentation

2D and 3D visual understanding with limited supervision

Contact Info

Product

Resources

About