Single Network Panoptic Segmentation for Street Scene Understanding

Abstract

In this work, we propose a single deep neural network for panoptic segmentation, for which the goal is to provide each individual pixel of an input image with a class label, as in semantic segmentation, as well as a unique identifier for specific objects in an image, following instance segmentation. Our network makes joint semantic and instance segmentation predictions and combines these to form an output in the panoptic format. This has two main benefits: firstly, the entire panoptic prediction is made in one pass, reducing the required computation time and resources; secondly, by learning the tasks jointly, information is shared between the two tasks, thereby improving performance. Our network is evaluated on two street scene datasets: Cityscapes and Mapillary Vistas. By leveraging information exchange and improving the merging heuristics, we increase the performance of the single network, and achieve a score of 23.9 on the Panoptic Quality (PQ) metric on Mapillary Vistas validation, with an input resolution of 640 x 900 pixels. On Cityscapes validation, our method achieves a PQ score of 45.9 with an input resolution of 512 x 1024 pixels. Moreover, our method decreases the prediction time by a factor of 2 with respect to separate networks.

Part-aware Panoptic Segmentation

“…Initially, most panoptic segmentation methods applied a multi-task network that trains and outputs instance segmentation and semantic segmentation in parallel, followed by a merging operation to generate panoptic segmentation results [11,24,32,46,49]. Recently, more methods are introduced that focus on optimizing the process of merging to panoptic segmentation [27,40,55,58], or try to solve the task more holistically or efficiently [7,12,21,31,57].…”

YOLO-based Panoptic Segmentation Network

“…And as can be seen in table III, using a smaller images increases inference speed by 2× at the expense of 3 Panoptic Quality points. PQ, SQ and RQ correspond to the Panoptic Quality, Architecture PQ(%) Time(ms) JSISNet [17] 17.6 n/a AUNet [5] 59.0 n/a Panoptic FPN [3] 58.1 n/a Single Network PS [18] 42.9 590 DeeperLab [8] 56.53 308 Panoptic Deeplab [9] 63.0 175 AdapIS [10] 62.0 n/a FPSNet [19] 55.1 114 Real-Time PS [20] 58…”

Hierarchical Lovász Embeddings for Proposal-free Panoptic Segmentation

“…They subsequently combine these using heuristics. Subsequent work was done to combine the independent networks into one, by adding a semantic segmentation branch to Mask R-CNN [23,43,13,29,26,27], but manual heuristics still remained. Other works aim to further remove manual merging heuristics of the semantics and instance predictions.…”

“…To improve upon this, previous works combine both tasks in a joint model [52,23,27]. Early methods focus more on a joint model and the majority of them leverage two-stage instance detection models [43,52,23,29,13]; some recent works propose bottom-up [15,9,53], yet instance and semantic segmentation are still treated separately. Performing panoptic segmentation as a single task without duplicated information across sub-tasks remains an interesting question.…”