PLIN: A Network for Pseudo-LiDAR Point Cloud Interpolation

Abstract

LiDAR sensors can provide dependable 3D spatial information at a low frequency (around 10Hz) and have been widely applied in the field of autonomous driving and UAV. However, the camera with a higher frequency (around 20Hz) has to be decreased so as to match with LiDAR in a multisensor system. In this paper, we propose a novel Pseudo-LiDAR interpolation network (PLIN) to increase the frequency of Li-DAR sensors. PLIN can generate temporally and spatially highquality point cloud sequences to match the high frequency of cameras. To achieve this goal, we design a coarse interpolation stage guided by consecutive sparse depth maps and motion relationship. We also propose a refined interpolation stage guided by the realistic scene. Using this coarse-to-fine cascade structure, our method can progressively perceive multi-modal information and generate accurate intermediate point clouds.To the best of our knowledge, this is the first deep framework for Pseudo-LiDAR point cloud interpolation, which shows appealing applications in navigation systems equipped with LiDAR and cameras. Experimental results demonstrate that PLIN achieves promising performance on the KITTI dataset, significantly outperforming the traditional interpolation method and the state-of-the-art video interpolation technique.

FPPN: Future Pseudo-LiDAR Frame Prediction for Autonomous Driving

“…Compared with video frame interpolation, three-dimensional frame interpolation is more challenging. To solve the low-frequency problem of the LiDAR, [7] utilizes a learning-based method to predict bi-directional optical flow to estimate the motion between consecutive frames. The two input frames are further warped and generate the intermediate depth frames through a coarse-to-fine cascade structure.…”

“…Therefore, there is a noticeable dislocation between the LiDAR and other sensors such as cameras(around 20Hz) in a multi-sensor system. To increase the frequency of the LiDAR, Liu et al [7] propose to interpolate an intermediate frame. However, the interpolation in two consecutive frames is a post-processing operation, which cannot be applied to high-speed practical application scenarios.…”

Pseudo-LiDAR Point Cloud Interpolation Based on 3D Motion Representation and Spatial Supervision

“…PLIN [5] is the first work to interpolate the intermediate Pseudo-LiDAR point cloud given two consecutive stereo pairs. It utilized a coarse-to-fine network structure to facilitate the perception of multi-modal information such as optical flow and color images.…”

“…The comparison of Pseudo-LiDAR point cloud interpolation methods. We visualize the interpolated depth maps and Pseudo-LiDAR point clouds obtained by PLIN [5] and our approach. The cropped region (d) indicates the ground truth point cloud from the sparse LiDAR data (a).…”

“…Due to the huge volume of the point cloud captured by LiDAR, directly processing and learning on 3D space is time-consuming. PLIN [5] presents the first pipeline for the Pseudo-LiDAR point cloud interpolation task, in which the Pseudo-LiDAR point cloud is obtained by the interpolated dense depth map and camera parameters. PLIN increases the frequency of LiDAR sensors by interpolating adjacent point clouds, to solve the problem of frequency mismatching between LiDAR and camera.…”