Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Jaus, Alexander; Yang, Kailun; Stiefelhagen, Rainer

doi:10.1109/iv48863.2021.9575904

Cited by 28 publications

(8 citation statements)

References 85 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, to address the dearth of annotated panoramic images, researchers have explicitly formalized panoramic segmentation as an unsupervised domain adaptation problem or a domain generalization problem by transferring from the data-rich pinhole domain to the data-scarce panoramic domain [253], [255]. For domain adaptation, one can use labeled pinhole data as the source domain and unlabeled panoramic images as the target domain.…”

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

“…11(c)) designed attentionaugmented domain adaptation modules to detect and magnify the pinhole-panoramic correspondences in multiple spaces. Jaus et al [255] introduced panoramic panoptic segmentation, which extended panoramic semantic segmentation by also offering instance-level understanding (Fig. 11(d)).…”

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

See 1 more Smart Citation

Review on Panoramic Imaging and Its Applications in Scene Understanding

Gao¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

With the rapid development of high-speed communication and artificial intelligence technologies, human perception of real-world scenes is no longer limited to the use of small Field of View (FoV) and low-dimensional scene detection devices. Panoramic imaging emerges as the next generation of innovative intelligent instruments for environmental perception and measurement. However, while satisfying the need for large-FoV photographic imaging, panoramic imaging instruments are expected to have high resolution, no blind area, miniaturization, and multi-dimensional intelligent perception, and can be combined with artificial intelligence methods towards the next generation of intelligent instruments, enabling deeper understanding and more holistic perception of 360 • real-world surrounding environments. Fortunately, recent advances in freeform surfaces, thinplate optics, and metasurfaces provide innovative approaches to address human perception of the environment, offering promising ideas beyond conventional optical imaging. In this review, we begin with introducing the basic principles of panoramic imaging systems, and then describe the architectures, features, and functions of various panoramic imaging systems. Afterwards, we discuss in detail the broad application prospects and great design potential of freeform surfaces, thin-plate optics, and metasurfaces in panoramic imaging. We then provide a detailed analysis on how these techniques can help enhance the performance of panoramic imaging systems. We further offer a detailed analysis of applications of panoramic imaging in scene understanding for autonomous driving and robotics, spanning panoramic semantic image segmentation, panoramic depth estimation, panoramic visual localization, and so on. Finally, we cast a perspective on future potential and research directions for panoramic imaging instruments.

show abstract

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

Section: A Semantic Scene Understanding With Image Segmentationmentioning

confidence: 99%

Review on Panoramic Imaging and Its Applications in Scene Understanding

Gao¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Mainstream outdoor omnidirectional semantic segmentation systems rely on fisheye cameras [57], [58], [59] or panoramic images [60], [61], [62]. Panoramic panoptic segmentation is also addressed in recent surrounding parsing systems [63], [64], [65], where the video segmentation pipeline with the Waymo open dataset [64] has a coverage of 220 • . Indoor methods, on the other hand, focus on either distortion-mitigated representations [66], [67], [68], [69], [70] or multi-tasks schemes [8], [71], [72].…”

Section: Panoramic Segmentationmentioning

confidence: 99%

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Zhang¹,

Yang²,

Sheng³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we address panoramic semantic segmentation, which provides a full-view and dense-pixel understanding of surroundings in a holistic way. Panoramic segmentation is under-explored due to two critical challenges: (1) image distortions and object deformations on panoramas; (2) lack of annotations for training panoramic segmenters. To tackle these problems, we propose a Transformer for Panoramic Semantic Segmentation (Trans4PASS) architecture. First, to enhance distortion awareness, Trans4PASS, equipped with Deformable Patch Embedding (DPE) and Deformable MLP (DMLP) modules, is capable of handling object deformations and image distortions whenever (before or after adaptation) and wherever (shallow or deep levels) by design. We further introduce the upgraded Trans4PASS+ model, featuring DMLPv2 with parallel token mixing to improve the flexibility and generalizability in modeling discriminative cues. Second, we propose a Mutual Prototypical Adaptation (MPA) strategy for unsupervised domain adaptation. Third, aside from Pinhole-to-Panoramic (PIN2PAN) adaptation, we create a new dataset (SynPASS) with 9,080 panoramic images to explore a Synthetic-to-Real (SYN2REAL) adaptation scheme in 360 • imagery. Extensive experiments are conducted, which cover indoor and outdoor scenarios, and each of them is investigated with PIN2PAN and SYN2REAL regimens. Trans4PASS+ achieves state-of-the-art performances on four domain adaptive panoramic semantic segmentation benchmarks. Code is available at https://github.com/jamycheung/Trans4PASS.

show abstract

“…"S": supervised, "U": Unsupervised, "D": domain adaptation. [121], [122] and [110]. Particularly, in [110], a shared attention module is used to extract features from the 2D domain and panoramic domain, and two domain adaption modules are used to "teach" the panoramic branch by the perspective branch.…”

Section: Semantic Segmentationmentioning

confidence: 99%

Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

Ai¹,

Cao²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

To whom it may concern,In this paper, we provide an in-depth and comprehensive survey on recent technical advancements in deep learning for omnidirectional vision, and also provide new perspectives for the future direction. Deep learning (DL) has recently been applied to omnidirectional vision. DL-based omnidirectional vision methods often achieve the state-of-the-art (SoTA) performance on various benchmark datasets. Diverse deep neural network (DNN) models have been developed, ranging from the convolutions neural network (CNN)-based models to vision transformers (ViT)-based ones. Generally, the SoTA DNN-based methods differ from each other in four major aspects: convolutional filters used to extract features from the omnidirectional image (ODI) data, network design considering the input numbers and projection types, omnidirectional vision with novel learning strategies, and practical applications. In this paper, we provide a comprehensive and systematic review and analysis of the recent progress in DL methods for omnidirectional vision. There are some previous surveys in the literature. However, some of them are focused on the specific vision tasks, especially room layout reconstruction, 3D scene geometry recovery. Moreover, some provide limited reviews of omnidirectional video streaming methods. Unlike existing surveys, this paper highlights the importance of deep learning and probe the recent advances for omnidirectional vision, both methodically and comprehensively. To the best of our knowledge, this is the first survey to comprehensively review and analyze the DL methods for omnidirectional vision. The main contributions of this paper to the community are five folds: (I) We comprehensively review and analyze the DL methods for omnidirectional vision, including the omnidirectional imaging principle, representation learning, datasets, a taxonomy, and applications, to highlight the differences and difficulties with the 2D planner image data. (II) We conduct an analytical study of recent trends of DL for omnidirectional vision, both hierarchically and structurally. Moreover, we offer insights into the discussion and challenge of each category. (III) We summarize the latest novel learning strategies and potential applications for omnidirectional vision.(I) We provide insightful discussions of the challenges and open problems yet to be solved and propose the potential future directions to spur more in-depth research by the community.(II) We create an open-source repository that provides a taxonomy of all the mentioned works and code links, and hope it can shed light on future research. The organization of this paper is structured as follows. In Sec.2, we introduce the imaging principle of ODI, convolution methods for omnidirectional vision, and some representative datasets. In Sec.3, we review and analyze the existing DL approaches for various tasks and provide taxonomies to categorize the relevant papers. Sec.4 covers novel learning paradigms for the tasks in omnidirectional vision, e.g., unsupervised learning, transfer learning, and reinforcement learning. Sec.5 then scrutinizes the applications, followed by Sec. 6, where we discuss open problems and future directions. Finally, we conclude this paper in Sec. 7. Furthermore, we summarize most, if not all but representative, works (over 200 papers) in the last five years, which were published in the top-tier conferences and journals in computer vision/graphics and machine learning. Due to the lack of space, we show the experimental results in Sec. 2 of the suppl. material. We believe that this paper will bring significant interest in such crucial topics in the community and provide fundamental technical references for further research.

show abstract

Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

Cited by 28 publications

References 85 publications

Review on Panoramic Imaging and Its Applications in Scene Understanding

Review on Panoramic Imaging and Its Applications in Scene Understanding

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation

Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

Contact Info

Product

Resources

About