A survey on Deep Learning Advances on Different 3D Data Representations

Abstract

3D data is a valuable asset the computer vision filed as it provides rich information about the full geometry of sensed objects and scenes. Recently, with the availability of both large 3D datasets and computational power, it is today possible to consider applying deep learning to learn specific tasks on 3D data such as segmentation, recognition and correspondence. Depending on the considered 3D data representation, different challenges may be foreseen in using existent deep learning architectures. In this work, we provide a comprehensive overview about various 3D data representations highlighting the difference between Euclidean and non-Euclidean ones. We also discuss how Deep Learning methods are applied on each representation, analyzing the challenges to overcome.Concepts: • General and references → Surveys and overviews; • Computing methodologies → 3D Deep Learning; 3D computer vision applications; 3D data representations;

A singular Riemannian geometry approach to Deep Neural Networks I. Theoretical foundations

“…with n ≥ 2 such that the manifolds M i abide to hypothesis (4) and the maps Λ i are as in definition (15).…”

“…Indeed, in many cases of practical interest, one must study data whose underlying structure is that of a non-Euclidean space, the most prominent case is that of data lying on a graph. Some examples are 2D meshed surfaces in computer graphics [15,16] or weighted graphs in social networks analysis [17]. To be concrete, consider a cloud of points which can be studied using a weighted graph.…”

Geometric Deep Learning on Molecular Representations

“…Each point on such a continuous surface is characterized by its chemical (e.g., hydrophobicity, electrostatics) and geometric features (e.g., shape, curvature). From a geometrical perspective, molecular surfaces are considered as 3D meshes, i.e., a set of polygons called faces described in terms of a set of vertices that describe how the mesh coordinates exist in the 3D space [115]. The vertices can be represented by a 2D grid structure (where four vertices on the mesh define a pixel) or by a 3D graph structure.…”

Dense graph convolutional neural networks on 3D meshes for 3D object segmentation and classification

“…As shown in Eq. 7, the feature of node i in the (l + 1)-th layer is not only H (l) , but also the set of node features of all previous layers {H (l) , H (l−1) , ..., H (1) }.…”

“…The main reason is that deep learning requires a large amount of training data while the cost of acquiring 3D data is much higher than that of 1D and 2D, and the representation of 3D data is much more complicated. Therefore, the application of deep learning in 3D is not as efficient in 2D [1]. Fortunately, with the development of 3D sensor technology, the cost of 3D data acquisition is getting lower and lower, which makes the application of deep learning increasingly popular in the 3D computer vision community.…”