Unifying Training and Inference for Panoptic Segmentation

Li, Qizhu; Qi, Xiaojuan; Torr, Philip H. S.

doi:10.1109/cvpr42600.2020.01333

Cited by 72 publications

(43 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Table 16, we conduct experiments on COCO val set. Compared with recent approaches, Panoptic FCN achieves superior performance with efficiency, which surpasses leading box-based [33] and box-free [39] methods over 0.2% and 1.5% PQ, respectively. With simple enhancement, the gap enlarges to 0.9% and 2.2% PQ.…”

Section: Cocomentioning

confidence: 92%

“…els[32],[33], the proposed method still achieves comparable result. If equipped with SwinT-based backbone, the proposed method achieves much better performance.…”

mentioning

confidence: 83%

“…From the view of instance representation, previous work mainly formats things and stuff from different perspectives. Foreground things are usually separated and represented with boxes [2], [31], [32], [33] or aggregated according to center offsets [5], while background stuff is often predicted with a parallel FCN [8] branch. Although methods of [6], [23] represent things and stuff uniformly, the inherent ambiguity cannot be resolved well merely with the pixel-level affinity, which yields the performance drop in complex scenarios.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision

Li¹,

Zhao²,

Qi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we present a conceptually simple, strong, and efficient framework for fully-and weakly-supervised panoptic segmentation, called Panoptic FCN. Our approach aims to represent and predict foreground things and background stuff in a unified fully convolutional pipeline, which can be optimized with point-based fully or weak supervision. In particular, Panoptic FCN encodes each object instance or stuff category with the proposed kernel generator and produces the prediction by convolving the high-resolution feature directly. With this approach, instance-aware and semantically consistent properties for things and stuff can be respectively satisfied in a simple generate-kernel-then-segment workflow. Without extra boxes for localization or instance separation, the proposed approach outperforms the previous box-based and -free models with high efficiency. Furthermore, we propose a new form of pointbased annotation for weakly-supervised panoptic segmentation. It only needs several random points for both things and stuff, which dramatically reduces the annotation cost of human. The proposed Panoptic FCN is also proved to have much superior performance in this weakly-supervised setting, which achieves 82% of the fully-supervised performance with only 20 randomly annotated points per instance. Extensive experiments demonstrate the effectiveness and efficiency of Panoptic FCN on COCO, VOC 2012, Cityscapes, and Mapillary Vistas datasets. And it sets up a new leading benchmark for both fully-and weakly-supervised panoptic segmentation. Our code and models are made publicly available at https://github.com/dvlab-research/PanopticFCN.

show abstract

Section: Cocomentioning

confidence: 92%

“…els[32],[33], the proposed method still achieves comparable result. If equipped with SwinT-based backbone, the proposed method achieves much better performance.…”

mentioning

confidence: 83%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision

Li¹,

Zhao²,

Qi³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Compared to bottom-up methods, we achieve comparable performance, but smaller inference time. Two-stage approaches [9,14,30] based on Mask R-CNN [8] provide the most accurate segmentation, having the highest PQ scores. At the same time, two-stage methods are the slowest.…”

Section: Performance On Cocomentioning

confidence: 99%

Fast Panoptic Segmentation with Soft Attention Embeddings

Petrovai

Nedevschi

2022

Sensors

View full text Add to dashboard Cite

Panoptic segmentation provides a rich 2D environment representation by unifying semantic and instance segmentation. Most current state-of-the-art panoptic segmentation methods are built upon two-stage detectors and are not suitable for real-time applications, such as automated driving, due to their high computational complexity. In this work, we introduce a novel, fast and accurate single-stage panoptic segmentation network that employs a shared feature extraction backbone and three network heads for object detection, semantic segmentation, instance-level attention masks. Guided by object detections, our new panoptic segmentation head learns instance specific soft attention masks based on spatial embeddings. The semantic masks for stuff classes and soft instance masks for things classes are pixel-wise coherent and can be easily integrated in a panoptic output. The training and inference pipelines are simplified and no post-processing of the panoptic output is necessary. Benefiting from fast inference speed, the network can be deployed in automated vehicles or robotic applications. We perform extensive experiments on COCO and Cityscapes datasets and obtain competitive results in both accuracy and time. On the Cityscapes dataset we achieve 59.7 panoptic quality with an inference speed of more than 10 FPS on high resolution 1024 × 2048 images.

show abstract

“…Early attempts [19] in PS follow the decomposition pipeline, separately predicting the semantic and instance segmentation results and then adopting the things-stuff fusion process in later stages. Several works try to simplify the process and improve the accuracy through replacing the post things-stuff fusion with the parameter-free [48] or trainable [23] panoptic head. Furthermore, more researchers [24,39] try to abandon the separated branches and build an end-to-end unified framework.…”

Section: Related Workmentioning

confidence: 99%

Slot-VPS: Object-centric Representation Learning for Video Panoptic Segmentation

Zhou¹,

Zhang²,

Lee³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Video Panoptic Segmentation (VPS) aims at assigning a class label to each pixel, uniquely segmenting and identifying all object instances consistently across all frames. Classic solutions usually decompose the VPS task into several sub-tasks and utilize multiple surrogates (e.g. boxes and masks, centers and offsets) to represent objects. However, this divide-and-conquer strategy requires complex postprocessing in both spatial and temporal domains and is vulnerable to failures from surrogate tasks. In this paper, inspired by object-centric learning which learns compact and robust object representations, we present Slot-VPS, the first end-to-end framework for this task. We encode all panoptic entities in a video, including both foreground instances and background semantics, with a unified representation called panoptic slots. The coherent spatio-temporal object's information is retrieved and encoded into the panoptic slots by the proposed Video Panoptic Retriever, enabling to localize, segment, differentiate, and associate objects in a unified manner. Finally, the output panoptic slots can be directly converted into the class, mask, and object ID of panoptic objects in the video. We conduct extensive ablation studies and demonstrate the effectiveness of our approach on two benchmark datasets, Cityscapes-VPS (val and test sets) and VIPER (val set), achieving new state-of-the-art performance of 63.7, 63.3 and 56.2 VPQ, respectively.

show abstract

Unifying Training and Inference for Panoptic Segmentation

Cited by 72 publications

References 17 publications

Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision

Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision

Fast Panoptic Segmentation with Soft Attention Embeddings

Slot-VPS: Object-centric Representation Learning for Video Panoptic Segmentation

Contact Info

Product

Resources

About