Sparse depth sensing for resource-constrained robots

Abstract

We consider the case in which a robot has to navigate in an unknown environment but does not have enough on-board power or payload to carry a traditional depth sensor (e.g., a 3D lidar) and thus can only acquire a few (point-wise) depth measurements. We address the following question: is it possible to reconstruct the geometry of an unknown environment using sparse and incomplete depth measurements? Reconstruction from incomplete data is not possible in general, but when the robot operates in man-made environments, the depth exhibits some regularity (e.g., many planar surfaces with only a few edges); we leverage this regularity to infer depth from a small number of measurements. Our first contribution is a formulation of the depth reconstruction problem that bridges robot perception with the compressive sensing literature in signal processing. The second contribution includes a set of formal results that ascertain the exactness and stability of the depth reconstruction in 2D and 3D problems, and completely characterize the geometry of the profiles that we can reconstruct. Our third contribution is a set of practical algorithms for depth reconstruction: our formulation directly translates into algorithms for depth estimation based on convex programming. In real-world problems, these convex programs are very large and general-purpose solvers are relatively slow. For this reason, we discuss ad-hoc solvers that enable fast depth reconstruction in real problems. The last contribution is an extensive experimental evaluation in 2D and 3D problems, including Monte Carlo runs on simulated instances and testing on multiple real datasets. Empirical results confirm that the proposed approach ensures accurate depth reconstruction, outperforms interpolation-based strategies, and performs well even when the assumption of structured environment is violated. SUPPLEMENTAL MATERIAL• Video demonstrations: https://youtu.be/vE56akCGeJQ• Source code: https://github.com/sparse-depth-sensing

Self-Supervised Sparse-to-Dense: Self-Supervised Depth Completion from LiDAR and Monocular Camera

“…Depth completion. Depth completion is an umbrella term that covers a collection of related problems with a variety of different input modalities (e.g., relatively dense depth input [5,6,7] vs. sparse depth measurements [8,9]; with color images for guidance [6,10] vs. without [4]). The problems and solutions are usually sensor-dependent, and as a result they face vastly different levels of algorithmic challenges.…”

“…However, the completion problem becomes much more challenging when the input depth image has much lower density, because the inverse problem is ill-posed. For instance, Ma et al [8,9] addressed depth reconstruction from only hundreds of depth measurements, by assuming a strong a priori of piecewise linearity in depth signals. Another example is autonomous driving with 3D LiDARs, where the projected depth measurements on the camera image space account for roughly 4% pixels [4].…”

“…To alleviate this issue, we add a third term to the loss functions in order to encourage smoothness of the depth predictions. Inspired by [9,8,28], we penalize ∇ 2 pred 1 1 , the L 1 loss of the second-order derivatives of the depth predictions, to encourage piecewise-linear depth signal.…”

AutoPCD: Learning-Augmented Indoor Point Cloud Completion

“…Plane Coefficients Prediction: AutoPCD then approximates the PCD into several 3D planes and learn the plane coefficients. The idea shares a similar spirit to a recently proposed work on sparse depth reconstruction [4]. However, different from the existing approach using compressed sensing and geometrical modeling, AutoPCD leverages machine-learning to learn the coefficients.…”

“…To make the feature extraction robust, AutoPCD transforms the original points from Cartesian coordinates into learned planes' coordinates: The Encoder module leverages the plane coefficients, predicted by the Plane Predictor, and multiply them with Cartesian point coordinates to achieve the transformation.Plane Coefficients Prediction: AutoPCD then approximates the PCD into several 3D planes and learn the plane coefficients. The idea shares a similar spirit to a recently proposed work on sparse depth reconstruction[4]. However, different from the existing approach using compressed sensing and geometrical modeling, AutoPCD leverages machine-learning to learn the coefficients.…”