ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation

Abstract

In this paper, we investigate if we could make the self-training -a simple but popular framework -work better for semi-supervised segmentation. Since the core issue in semi-supervised setting lies in effective and efficient utilization of unlabeled data, we notice that increasing the diversity and hardness of unlabeled data is crucial to performance improvement. Being aware of this fact, we propose to adopt the most plain self-training scheme coupled with appropriate strong data augmentations on unlabeled data (namely ST) for this task, which surprisingly outperforms previous methods under various settings without any bells and whistles. Moreover, to alleviate the negative impact of the wrongly pseudo labeled images, we further propose an advanced self-training framework (namely ST++), that performs selective re-training via selecting and prioritizing the more reliable unlabeled images. As a result, the proposed ST++ boosts the performance of semi-supervised model significantly and surpasses existing methods by a large margin on the Pascal VOC 2012 and Cityscapes benchmark. Overall, we hope this straightforward and simple framework will serve as a strong baseline or competitor for future works.Code is available at https://github.com/LiheYoung/ST-PlusPlus.

Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels

“…Existing supervised approaches rely on large-scale annotated data, which can be too costly to acquire in practice. To alleviate this problem, many attempts [1,4,9,15,21,33,43,48] have been made towards semi-supervised semantic segmentation, which learns a model with only a few labeled samples and numerous unlabeled ones. Under such a setting, how to adequately leverage the unlabeled data becomes critical.…”

“…Concretely, given an unlabeled image, prior arts [27,41] borrow predictions from the model trained on labeled data, and use the pixel-wise prediction as the "ground-truth" to in turn boost the supervised model. To mitigate the problem of confirmation bias [2], where the model may suffer from incorrect pseudo-labels, existing approaches propose to filter the predictions with their confidence scores [42,43,50,51]. In other words, only the highly confident predictions are used as the pseudo-labels, while the ambiguous ones are discarded.…”

“…Recently, a more intuitive but effective framework: self-training [27], has become the mainstream. Several methods [15,43,44] utilize strong data augmentation such as CutOut [13], CutMix [45], and ClassMix [31] based on self-training. However, these methods do not pay much attention to the characteristics of semantic segmentation, while our method focuses on those unreliable pixels which will be filtered out by most of selftraining based methods [34,43,44].…”

“…Several methods [15,43,44] utilize strong data augmentation such as CutOut [13], CutMix [45], and ClassMix [31] based on self-training. However, these methods do not pay much attention to the characteristics of semantic segmentation, while our method focuses on those unreliable pixels which will be filtered out by most of selftraining based methods [34,43,44]. Pseudo-Labeling is applied to prevent overfitting to incorrect pseudo-labels when generating predictions of input images from the teacher network [2,27].…”

Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation

“…Even though L2 loss is known to be robust, which is advantageous when dealing with the noisy predictions produced by consistencybased methods, it is also known to have poor converge and to possibly lead to vanishing gradients. Given the reliability of the segmentation predictions produced by our extended MT model, we instead use the more effective cross entropy (CE) loss, constrained to be computed at regions of high-confidence segmentation results, represented by c(ω) in (3), following the strategy applied in self-training approaches [18,39,41].…”