KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D

Liao, Yiyi; Xie, Jun; Geiger, Andreas

doi:10.48550/arxiv.2109.13410

Cited by 18 publications

(27 citation statements)

References 91 publications

(185 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluate our method on KITTI-odometry [25], KITTIdetection [25] and KITTI-360 [26] datasets, in which synchronized Velodyne point clouds and RGB images are collected from different scenes. The input point cloud and image pairs are aligned according to the timestamps.…”

Section: Methodsmentioning

confidence: 99%

DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow

Jing¹,

Ding²,

Xiong³

et al. 2022

Preprint

View full text Add to dashboard Cite

Accurate LiDAR-camera extrinsic calibration is a precondition for many multi-sensor systems in mobile robots. Most calibration methods rely on laborious manual operations and calibration targets. While working online, the calibration methods should be able to extract information from the environment to construct the cross-modal data association. Convolutional neural networks (CNNs) have powerful feature extraction ability and have been used for calibration. However, most of the past methods solve the extrinsic as a regression task, without considering the geometric constraints involved. In this paper, we propose a novel end-to-end extrinsic calibration method named DXQ-Net, using a differentiable pose estimation module for generalization. We formulate a probabilistic model for LiDARcamera calibration flow, yielding a prediction of uncertainty to measure the quality of LiDAR-camera data association. Testing experiments illustrate that our method achieves a competitive with other methods for the translation component and state-ofthe-art performance for the rotation component. Generalization experiments illustrate that the generalization performance of our method is significantly better than other deep learningbased methods.

show abstract

Section: Methodsmentioning

confidence: 99%

DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow

Jing¹,

Ding²,

Xiong³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We extend the KITTI-360 [18] dataset which has semantic and instance labels with amodal panoptic annotations and name it the KITTI-360-APS dataset. It consists of nine sequences of urban street scenes with annotations for 61,168 images of resolution 1408 × 376 pixels.…”

Section: Kitti-360-apsmentioning

confidence: 99%

Amodal Panoptic Segmentation

Mohan¹,

Valada²

2022

Preprint

View full text Add to dashboard Cite

Humans have the remarkable ability to perceive objects as a whole, even when parts of them are occluded. This ability of amodal perception forms the basis of our perceptual and cognitive understanding of our world. To enable robots to reason with this capability, we formulate and propose a novel task that we name amodal panoptic segmentation. The goal of this task is to simultaneously predict the pixelwise semantic segmentation labels of the visible regions of stuff classes and the instance segmentation labels of both the visible and occluded regions of thing classes. To facilitate research on this new task, we extend two established benchmark datasets with pixel-level amodal panoptic segmentation labels that we make publicly available as KITTI-360-APS and BDD100K-APS. We present several strong baselines, along with the amodal panoptic quality (APQ) and amodal parsing coverage (APC) metrics to quantify the performance in an interpretable manner. Furthermore, we propose the novel amodal panoptic segmentation network (APSNet), as a first step towards addressing this task by explicitly modeling the complex relationships between the occluders and occludes. Extensive experimental evaluations demonstrate that APSNet achieves state-of-the-art performance on both benchmarks and more importantly exemplifies the utility of amodal recognition. The benchmarks are available at http://amodal-panoptic.cs.unifreiburg.de.

show abstract

“…PASS [14] presents a panoramic annular semantic segmentation framework with an associated dataset for credible evaluation. KITTI-360 [31] is collected with perspective stereo cameras, a pair of fisheye cameras, and a laser scanning unit for enabling 360 • perception. WoodScape [32] comprises of multiple surround-view fisheye cameras and multiple tasks like segmentation and soiling detection.…”

Section: B Panoramic Scene Understanding Datasetsmentioning

confidence: 99%

PanoFlow: Learning 360° Optical Flow for Surrounding Temporal Understanding

Sheng¹,

Zhou²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Optical flow estimation is a basic task in selfdriving and robotics systems, which enables to temporally interpret the traffic scene. Autonomous vehicles clearly benefit from the ultra-wide Field of View (FoV) offered by 360 • panoramic sensors. However, due to the unique imaging process of panoramic images, models designed for pinhole images do not directly generalize satisfactorily to 360 • panoramic images. In this paper, we put forward a novel network framework--PANOFLOW, to learn optical flow for panoramic images. To overcome the distortions introduced by equirectangular projection in panoramic transformation, we design a Flow Distortion Augmentation (FDA) method. We further propose a Cyclic Flow Estimation (CFE) method by leveraging the cyclicity of spherical images to infer 360 • optical flow and converting large displacement to relatively small displacement. PanoFlow is applicable to any existing flow estimation method and benefit from the progress of narrow-FoV flow estimation. In addition, we create and release a synthetic panoramic dataset Flow360 based on CARLA to facilitate training and quantitative analysis. PanoFlow achieves state-of-the-art performance. Our proposed approach reduces the End-Point-Error (EPE) on the established Flow360 dataset by 26%. On the public OmniFlowNet dataset, PanoFlow achieves an EPE of 3.34 pixels, a 53.1% error reduction from the best published result (7.12 pixels). We also validate our method via an outdoor collection vehicle, indicating strong potential and robustness for real-world navigation applications. Code and dataset are publicly available at https://github.com/MasterHow/PanoFlow.

show abstract

KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D

Cited by 18 publications

References 91 publications

DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow

DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow

Amodal Panoptic Segmentation

PanoFlow: Learning 360° Optical Flow for Surrounding Temporal Understanding

Contact Info

Product

Resources

About