Frame-to-Frame Aggregation of Active Regions in Web Videos for Weakly Supervised Semantic Segmentation

Abstract

When a deep neural network is trained on data with only image-level labeling, the regions activated in each image tend to identify only a small region of the target object. We propose a method of using videos automatically harvested from the web to identify a larger region of the target object by using temporal information, which is not present in the static image. The temporal variations in a video allow different regions of the target object to be activated. We obtain an activated region in each frame of a video, and then aggregate the regions from successive frames into a single image, using a warping technique based on optical flow. The resulting localization maps cover more of the target object, and can then be used as proxy ground-truth to train a segmentation network. This simple approach outperforms existing methods under the same level of supervision, and even approaches relying on extra annotations. Based on VGG-16 and ResNet 101 backbones, our method achieves the mIoU of 65.0 and 67.4, respectively, on PASCAL VOC 2012 test images, which represents a new state-of-the-art.

Box2Seg: Attention Weighted Loss and Discriminative Feature Learning for Weakly Supervised Segmentation

“…Previous works on weakly-supervised semantic segmentation have used image-level annotations [17,20,26,27,42,50,52], points/clicks [4], scribbles [29,47,48,49], bounding box annotations [11,19,37,41,46,55] and adversarial training [3,21]. We take a closer look at some of these methods and categorize them based on the labels required and their methodology.…”

“…Our segmentation network architecture is similar to UPerNet [51] where the encoder backbone is ResNet-101 [16], and decoders consist of 2 convolutional layers. We employ the ResNet-101 backbone to ensure fair comparison with the three most recent SOTA works, SDI [23], Li et al [28], and BCM [46] as well as 4 other recent methods [27,47,48,49] in Table 1. We have three decoders, one each for the y, α, and β branches.…”

“…Weakly supervised approaches using bounding box annotations have been shown to be more accurate when compared to methods that use only image-level labels. SOTA segmentation result on VOC using bounding box annotations [46] outperforms the SOTA methods using image-level labels [27] by ∼ 4%. Our work, Box2Seg, builds upon previous approaches that use bounding box annotations.…”

Deep graph cut network for weakly-supervised semantic segmentation

“…Various weak annotations have been adopted in this research field, such as bounding boxes [17][18][19], scribbles [20], points [21], and image-level labels [5,11,13]. Moreover, some research studies, e.g., [22,23], improve the performance with additional and unlabeled data. Usually, the data are obtained from the Internet called web data.…”

“…Usually, the data are obtained from the Internet called web data. So these are also called weblysupervised segmentation methods [22]. In this paper, we utilize the image-level labels which are very cheap to obtain and do not provide any localization information about the object in the image.…”