Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integration

Yang, Zongxin; Wei, Yunchao; Yang, Yi

doi:10.1109/tpami.2021.3081597

Cited by 83 publications

(51 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Generative learning CFBI (TPAMI 2021) [35] ✗ ✗ ✓ ✗ CRW (NIPS 2020) [11] ✓ ✗ ✗ ✗ VFS (ICCV 2021) [31] ✓ ✗ ✗ ✗ DUL (NIPS 2021) […”

Section: Pixel-wise Informationmentioning

confidence: 99%

See 1 more Smart Citation

In-N-Out Generative Learning for Dense Unsupervised Video Segmentation

Xiao¹,

Peike²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we focus on the unsupervised Video Object Segmentation (VOS) task which learns visual correspondence from unlabeled videos. Previous methods are mainly based on the contrastive learning paradigm, which optimize either in pixel level or image level and show unsatisfactory scalability. Image-level optimization learns pixel-wise information implicitly therefore is sub-optimal for such dense prediction task, while pixel-level optimization ignores the high-level semantic scope for capturing object deformation. To complementarily learn these two levels of information in an unified framework, we propose the In-aNd-Out (INO) generative learning from a purely generative perspective, which captures both high-level and fine-grained semantics by leveraging the structural superiority of Vision Transformer (ViT) and achieves better scalability. Specifically, the in-generative learning recovers the corrupted parts of an image via inferring its fine-grained semantic structure, while the out-generative learning captures high-level semantics by imagining the global information of an image given only random fragments. To better discover the temporal information, we additionally force the inter-frame consistency from both feature level and affinity matrix level. Extensive experiments on DAVIS-2017 val and YouTube-VOS 2018 val show that our INO outperforms previous state-of-the-art methods by significant margins.

show abstract

“…Generative learning CFBI (TPAMI 2021) [35] ✗ ✗ ✓ ✗ CRW (NIPS 2020) [11] ✓ ✗ ✗ ✗ VFS (ICCV 2021) [31] ✓ ✗ ✗ ✗ DUL (NIPS 2021) […”

Section: Pixel-wise Informationmentioning

confidence: 99%

“…VOS via Self-supervised Learning. Some previous approaches [15,18,20,33,35] learn visual representation fully supervised by annotated datasets with pixel-wise labels. However, these fully-supervised VOS methods are highly restricted by the diversity of annotated categories and the scale of the annotated dataset.…”

Section: Related Workmentioning

confidence: 99%

In-N-Out Generative Learning for Dense Unsupervised Video Segmentation

Xiao¹,

Peike²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…As a result, methods with strong spatial constraints like KMN [21] or CFBI [9] typically perform better (while still suffering a drop in performance from validation to test-dev). Some would use additional engineering, such as using 600p videos instead of the standard 480p [17,21,76], or add spatial constraints [20].…”

Section: B Davis Test-devmentioning

confidence: 99%

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Cheng¹,

Tai²,

Tang³

2021

Preprint

View full text Add to dashboard Cite

This paper presents a simple yet effective approach to modeling space-time correspondences in the context of video object segmentation. Unlike most existing approaches, we establish correspondences directly between frames without reencoding the mask features for every object, leading to a highly efficient and robust framework. With the correspondences, every node in the current query frame is inferred by aggregating features from the past in an associative fashion. We cast the aggregation process as a voting problem and find that the existing inner-product affinity leads to poor use of memory with a small (fixed) subset of memory nodes dominating the votes, regardless of the query. In light of this phenomenon, we propose using the negative squared Euclidean distance instead to compute the affinities. We validated that every memory node now has a chance to contribute, and experimentally showed that such diversified voting is beneficial to both memory efficiency and inference accuracy. The synergy of correspondence networks and diversified voting works exceedingly well, achieves new state-of-the-art results on both DAVIS and YouTubeVOS datasets while running significantly faster at 20+ FPS for multiple objects without bells and whistles.

show abstract

“…Seong et al [53] developed a kernelized memory network and used the Hide-and-Seek strat- egy training to handle occlusions and segment boundary extraction. Yang et al [77] combined collaborative foreground-background integration with multi-scale matching to be robust to various object scales.…”

Section: End-to-end Temporal Learningmentioning

confidence: 99%

Contextual Guided Segmentation Framework for Semi-supervised Video Instance Segmentation

Le,

Nguyen,

Tran

2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose Contextual Guided Segmentation (CGS) framework for video instance segmentation in three passes. In the first pass, i.e. preview segmentation, we propose Instance Re-Identification Flow to estimate main properties of each instance (i.e., human/non-human, rigid/deformable, known/unknown category) by propagating its preview mask to other frames. In the second pass, i.e. contextual segmentation, we introduce multiple contextual segmentation schemes. For human instance, we develop skeleton-guided segmentation in a frame along with object flow to correct and refine the result across frames. For non-human instance, if the instance has a wide variation in appearance and belongs to known categories (which can be inferred from the initial mask), we adopt instance segmentation. If the non-human instance is nearly rigid, we train FCNs on synthesized images from the first frame of a video sequence. In the final pass, i.e. guided segmentation, we develop a novel fined-grained segmentation method on non-rectangular regions of interest (ROIs). The natural-shaped ROI is generated by applying guided attention from the neighbor frames of the current one to reduce the ambiguity in the segmentation of different overlapping instances. Forward mask propagation is followed by backward mask propagation to further restore missing instance fragments due to reappeared instances, fast motion, occlusion, or heavy deformation. Finally, instances in each frame are merged based on their depth values, together with human and nonhuman object interaction and rare instance priority. Experiments conducted on the DAVIS Test-Challenge dataset demonstrate the effectiveness of our proposed framework. We achieved the 3 rd consistently in the DAVIS Challenges 2017-2019 with

show abstract

Collaborative Video Object Segmentation by Multi-Scale Foreground-Background Integration

Cited by 83 publications

References 55 publications

In-N-Out Generative Learning for Dense Unsupervised Video Segmentation

In-N-Out Generative Learning for Dense Unsupervised Video Segmentation

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Contextual Guided Segmentation Framework for Semi-supervised Video Instance Segmentation

Contact Info

Product

Resources

About