Review: Deep Learning on 3D Point Clouds

Abstract

A point cloud is a set of points defined in a 3D metric space. Point clouds have become one of the most significant data formats for 3D representation and are gaining increased popularity as a result of the increased availability of acquisition devices, as well as seeing increased application in areas such as robotics, autonomous driving, and augmented and virtual reality. Deep learning is now the most powerful tool for data processing in computer vision and is becoming the most preferred technique for tasks such as classification, segmentation, and detection. While deep learning techniques are mainly applied to data with a structured grid, the point cloud, on the other hand, is unstructured. The unstructuredness of point clouds makes the use of deep learning for its direct processing very challenging. This paper contains a review of the recent state-of-the-art deep learning techniques, mainly focusing on raw point cloud data. The initial work on deep learning directly with raw point cloud data did not model local regions; therefore, subsequent approaches model local regions through sampling and grouping. More recently, several approaches have been proposed that not only model the local regions but also explore the correlation between points in the local regions. From the survey, we conclude that approaches that model local regions and take into account the correlation between points in the local regions perform better. Contrary to existing reviews, this paper provides a general structure for learning with raw point clouds, and various methods were compared based on the general structure. This work also introduces the popular 3D point cloud benchmark datasets and discusses the application of deep learning in popular 3D vision tasks, including classification, segmentation, and detection.

Vision-Guided Automation and Assistance

“…Being able to use the raw point cloud was an attractive option for the vision pipeline since it limited the additional computation needed to format the point cloud to pass it through the DNN. However, passing raw point clouds through the DNN presented a few challenges [1]. Point clouds are unstructured since they do not fall on a grid like 2D images.…”

“…Point clouds can also be sparse and irregularly distributed such that you may have more points in one area of the cloud than another. To help overcome these challenges, we chose to implement a DNN based on the PointNet architecture, which was considered the foundation for DNNs that use raw point clouds [1]. The PointNet architecture consisted mainly of multilayer perceptrons which transformed the feature's dimensionality from 3 (x, y, z coordinates), to 1024 with shared learnable parameters for each layer.…”

KVGCN: A KNN Searching and VLAD Combined Graph Convolutional Network for Point Cloud Segmentation

“…They all directly deal with the unordered point clouds, and learn global and local features to realize the classification and segmentation. These works have achieved excellent results, yet difficulties still exist in the processing of point clouds [21]. The effective local geometric feature correspondence cannot be established, and the information between the points cannot be well utilized, which results in low accuracy of semantic segmentation.…”

A Benchmark for Large-Scale Heritage Point Cloud Semantic Segmentation

“…In this scenario, the comeback of Deep Learning (DL) in several research fields has been overwhelming (Griffiths and Boehm, 2019). Deep Neural Networks (DNNs) settled as the more efficient technology for learning-based tasks (Paolanti et al, 2019;Bello et al, 2020). However, despite DNNs proved to be very promising for handling and recognising 3D data , for CH, manual operations look more trustworthy, at least to capture the real estate from point clouds (Murtiyoso and Grussenmeyer, 2019).…”