Benchmarking Deep Learning Models on Point Cloud Segmentation

Abstract

Point clouds are currently used for a variety of applications, such as detection tasks in medical and geological domains. Intelligent analysis of point clouds is considered a highly computationally demanding and challenging task, especially the segmentation task among the points. Although numerous deep learning models have recently been proposed to segment point cloud data, there is no clear instruction of which exactly neural network to utilize and then incorporate into a system dealing with point cloud segmentation analysis. Besides, the majority of the developed models emphasize more on the accuracy rather than the efficiency, in order to achieve great results. Consequently, the training, validation and testing phases of the models require a great number of processing hours and a huge amount of memory. These high computational requirements are commonly difficult to deal with for many users. In this article, we analyse five state-of-the-art deep learning models for part segmentation task and give meaningful insights into the utilization of each one. We advance guidelines based on different properties, considering both learning-related metrics, such as accuracy, and system-related metrics, such as run time and memory footprint. We further propose and analyse generalized performance metrics, which facilitate the model evaluation phase in segmentation tasks allowing users to select the most appropriate approach for their context in terms of accuracy and efficiency.

Effective Early Stopping of Point Cloud Neural Networks

“…However, research must go beyond pure accuracy-metrics and another open issue of utmost importance is the efficiency of such deep learning models dealing with 3D point clouds, although it is still in its early steps of research. Recent studies highlight that the efficiency of 3D point cloud segmentation models is a serious concern for the community [18,20,6]. However, the majority of new and advanced deep learning models emphasize on the improvement of segmentation accuracy, providing almost no information on the models' efficiency [8,10,13,14].…”

“…We can observe that the models stopped in the proposed stop-windows achieve approximately the same ImIoU values as the best achieved in whole learning process of 200 epochs. For instance, it can be seen that the learning of KPConv model can be stopped at any epoch inside the window of Swindow = According to a recent performance benchmark shown in [20], KPConv needs a great amount of time to complete the learning process on ShapeNet dataset and specifically more time than its competitors.…”

“…Different NN architectures have been proposed recently to segment inner structures of point cloud [12,13,15]. However, recent studies show that training and evaluating these models are a greatly time-consuming task and, in some cases, the number of epochs and time spend is not proportional to the accuracy achieved [20,19]. Indeed, in this process, it is important not just to solely upgrade the accuracy-related metrics of the learning, such as accuracy, precision, recall or F1-score, but also to achieve a balance between efficiency and accuracy [20].…”

Object Segmentation of Cluttered Airborne LiDAR Point Clouds

“…In this work, we select PointNet architecture as the base of our framework due to its simplicity, robustness and low execution time [13]. The goal of our work is to provide an end-to-end framework for real-world LiDAR applications.…”