Self-Supervised Video Object Segmentation by Motion-Aware Mask Propagation

Abstract

We propose a self-supervised spatio-temporal matching method, coined Motion-Aware Mask Propagation (MAMP), for video object segmentation. MAMP leverages the frame reconstruction task for training without the need for annotations. During inference, MAMP extracts high-resolution features from each frame to build a memory bank from the features as well as the predicted masks of selected past frames. MAMP then propagates the masks from the memory bank to subsequent frames according to our proposed motion-aware spatio-temporal matching module to handle fast motion and long-term matching scenarios. Evaluation on DAVIS-2017 and YouTube-VOS datasets show that MAMP achieves stateof-the-art performance with stronger generalization ability compared to existing self-supervised methods, i.e., 4.2% higher mean J &F on DAVIS-2017 and 4.85% higher mean J &F on the unseen categories of YouTube-VOS than the nearest competitor. Moreover, MAMP performs at par with many supervised video object segmentation methods.

Key points trajectory and multi-level depth distinction based refinement for video mirror and glass segmentation

“…Tan et al [30] propose a stepwise attention emphasis transformer for polyp segmentation, which combines convolutional layers with a transformer encoder to enhance both global and local feature extraction. Miao et al [31] utilizes a fast object motion tracker to predict regions of interest (ROIs) for the next frame, and proposes motion path memory to filter out redundant context by memorizing features within the motion path of objects between two frames. GFA [46] significantly improves the generalization capability of video object segmentation models by addressing both scene and semantic shifts, by leveraging frequency domain transformations and online feature updates.…”

“…While image-based segmentation techniques for glass and mirrors exist [12,24,25], their direct application to video sequences yields unsatisfactory results, making them impractical for real-world applications. Furthermore, conventional video object segmentation (VOS) methods without suitable video mirror and glass dataset fail to consider the unique motion characteristics associated with high-reflectivity materials like glass and mirrors [26][27][28][29][30][31]. Due to lack of video mirror and glass dataset and no consideration on reflective features, general deep-learning based VOS methods always tend to segment the reflections in mirrors or glass as real objects, resulting in inaccurate segmentation [32][33][34][35][36].…”

VAPCNet: Viewpoint-Aware 3D Point Cloud Completion

“…Incomplete shapes in 3D scans resulting from occlusion (both self-occlusion and occlusion by other objects) and low resolution of sensors often make them unsuitable for direct use in practical applications such as object grasping and Virtual Reality (VR) [16,9,25,10]. To remedy this, shape completion aims to recover complete 3D shapes from partial 3D scans.…”