Mask Transfiner for High-Quality Instance Segmentation

Abstract

The recent Segment Anything Model (SAM) represents a big leap in scaling up segmentation models, allowing for powerful zero-shot capabilities and flexible prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction quality falls short in many cases, particularly when dealing with objects that have intricate structures. We propose HQ-SAM, equipping SAM with the ability to accurately segment any object, while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability. Our careful design reuses and preserves the pre-trained model weights of SAM, while only introducing minimal additional parameters and computation. We design a learnable High-Quality Output Token, which is injected into SAM's mask decoder and is responsible for predicting the high-quality mask. Instead of only applying it on mask-decoder features, we first fuse them with early and final ViT features for improved mask details. To train our introduced learnable parameters, we compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is only trained on the introduced detaset of 44k masks, which takes only 4 hours on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 9 diverse segmentation datasets across different downstream tasks, where 7 out of them are evaluated in a zero-shot transfer protocol. Our code and models will be released at https://github.com/SysCV/SAM-HQ.

Video Mask Transfiner for High-Quality Video Instance Segmentation

“…Video Instance Segmentation (VIS) Existing VIS methods can be summarized into three categories: twostage, one-stage, and transformer-based. Two-stage approaches [2,19,29,30,62] extend the Mask R-CNN family [12,20] by designing an additional tracking branch for object association. One-stage works [4,27,32,63] adopt anchor-free detectors [50], generally using linear masks basis combination [3] or conditional mask prediction generation [49].…”

“…MaskFreeVIS achieves competitive VIS performance without using any video masks or even image mask labels on all datasets. Validated on various methods and backbones, MaskFreeVIS achieves 91.25% performance of its fully supervised counterparts, even outperforming a few recent fully-supervised methods [11,16,19,60] on the popular YTVIS benchmark. Our simple yet effective design greatly narrows the performance gap between weakly-supervised and fully-Table 1.…”

AISFormer: Amodal Instance Segmentation with Transformer

“…However, with the occlusion augmentation added to the object, the visible ground truth is treated as the amodal ground truth. In the second training stage, the amodal segmentations outputs from the trained UNet in the first phase are taken as a pseudo-ground truth for learning a standard instance segmenter -Mask R-CNN [24]. In the inference phase, Mask R-CNN trained on the generated amodal ground truth is expected to yield the correct AIS.…”

“…In QueryInst, queries are shared between tasks of detection and segmentation in each stage via dynamic convolutions. Unlike the previous works which process on regular dense tensors, Mask Tranfiner [24] first decomposes and represents an image region as a hierarchical quadtree. Then, all points on the quadtree are transformed into a query sequence to predict labels.…”

Beyond Semantic to Instance Segmentation: Weakly-Supervised Instance Segmentation via Semantic Knowledge Transfer and Self-Refinement

“…Recently proposed instance segmentation methods [5,8,9,13,15,16,24,33,43,47] have achieved remarkable performance owing to the availability of abundant of segmentation labels for training. However, compared to other label types (e.g., bounding box or point), segmentation labels necessitate delicate pixel-level annotations, demanding much more monetary cost and human effort.…”