Fast Panoptic Segmentation with Soft Attention Embeddings

Abstract

Panoptic segmentation provides a rich 2D environment representation by unifying semantic and instance segmentation. Most current state-of-the-art panoptic segmentation methods are built upon two-stage detectors and are not suitable for real-time applications, such as automated driving, due to their high computational complexity. In this work, we introduce a novel, fast and accurate single-stage panoptic segmentation network that employs a shared feature extraction backbone and three network heads for object detection, semantic segmentation, instance-level attention masks. Guided by object detections, our new panoptic segmentation head learns instance specific soft attention masks based on spatial embeddings. The semantic masks for stuff classes and soft instance masks for things classes are pixel-wise coherent and can be easily integrated in a panoptic output. The training and inference pipelines are simplified and no post-processing of the panoptic output is necessary. Benefiting from fast inference speed, the network can be deployed in automated vehicles or robotic applications. We perform extensive experiments on COCO and Cityscapes datasets and obtain competitive results in both accuracy and time. On the Cityscapes dataset we achieve 59.7 panoptic quality with an inference speed of more than 10 FPS on high resolution 1024 × 2048 images.

Mask-Pyramid Network: A Novel Panoptic Segmentation Method

“…We compare the performance of Mask-PNet with several representative methods from different architectures, such as Weakly Supervised [ 34 ], Panoptic FPN [ 13 ], AUNet [ 28 ], UPSNet [ 6 ], Seamless [ 35 ], PCV [ 12 ], Mask RCNN [ 4 ], SSAP [ 36 ], AttentionPS [ 37 ], and MaskConver [ 38 ]. Table 1 shows the results of the Cityscapes validation set.…”

“…In this sub-section, we compare the performance of our proposed Mask-PNet method with several existing state-of-the-art panoptic segmentation techniques. AUNet [ 28 ], Panoptic FPN [ 13 ], AdaptIS [ 37 ], UPSNet [ 6 ], Seamless [ 35 ], Mask RCNN [ 4 ], SSAP [ 36 ], AttentionPS [ 37 ], and MaskConver [ 38 ] are among the representative methods we compare against. To evaluate the performance of these methods, we utilize the COCO validation dataset and present the results in Table 2 .…”