Occluded Video Instance Segmentation: A Benchmark

Abstract

Can our video understanding systems perceive objects when a heavy occlusion exists in a scene? To answer this question, we collect a large-scale dataset called OVIS for occluded video instance segmentation, that is, to simultaneously detect, segment, and track instances in occluded scenes. OVIS consists of 296k high-quality instance masks from 25 semantic categories, where object occlusions usually occur. While our human vision systems can understand those occluded instances by contextual reasoning and association, our experiments suggest that current video understanding systems cannot. On the OVIS dataset, the highest AP achieved by state-of-the-art algorithms is only 16.3, which reveals that we are still at a nascent stage for understanding objects, instances, and videos in a real-world scenario. We also present a simple plug-and-play module that performs temporal feature calibration to complement missing object cues caused by occlusion. Built upon MaskTrack R-CNN and SipMask, we obtain a remarkable AP improvement on the OVIS dataset. The OVIS dataset and project code are available at http://songbai.site/ovis.

5th Place Solution for YouTube-VOS Challenge 2022: Video Object Segmentation

“…On the other hand, video data in the real world have additional temporal information compared to static image data, and the data form in the testing stage is video, so it is more straightforward to bring more video segmentation datasets to improve performance. Benefit from the release of several new datasets in the video segmentation field recently, such as YoutubeVIS which has more objects in each video, OVIS [18] which occlusion scenarios are significant, and VSPW [12]) which have dense annotations and high-quality resolution, we introduce them into the second training stage, thus significantly improving the performance of models.…”

“…In the pre-training stage, several static image datasets including COCO [9], ECSSD [19], MSRA10K [4], PASCAL-S [7], PASCAL-VOC [6] are used for preliminarily semantic learning. During the main training, video datasets including Youtube-VOS [23], DAVIS 2017 [17], YouTubeVIS [8], OVIS [18], and VSPW [12] are used to enhance the generalization and robustness of the model.…”

“…Inspired by these existing methods, we propose a simple yet effective solution for VOS, as shown in Figure 2. In order to deal with existing difficulties, we first analyze the Youtube-VOS dataset and other video segmentation related datasets (e.g., OVIS [18] and VSPW [12]). We find that other datasets can supplement the diversified scenes with similar objects and occlusion situation from the data aspect.…”

Amodal Cityscapes: A New Dataset, its Generation, and an Amodal Semantic Segmentation Challenge Baseline

“…Amodal perception is of interest in many application fields of computer vision. Hence, datasets for amodal perception can be found in different fields, e.g., instance [17] and video instance segmentation [19], [20], human recognition and deocclusion [21]. The OVIS dataset [19] provides instance masks for videos while additionally labeling the occlusion level of each instance.…”

“…Hence, datasets for amodal perception can be found in different fields, e.g., instance [17] and video instance segmentation [19], [20], human recognition and deocclusion [21]. The OVIS dataset [19] provides instance masks for videos while additionally labeling the occlusion level of each instance. SAIL-VOS [20] is a synthetic video instance segmentation dataset with amodal instance segmentation masks.…”