Learning Point Embeddings from Shape Repositories for Few-Shot Segmentation

Abstract

User generated 3D shapes in online repositories contain rich information about surfaces, primitives, and their geometric relations, often arranged in a hierarchy. We present a framework for learning representations of 3D shapes that reflect the information present in this meta data and show that it leads to improved generalization for semantic segmentation tasks. Our approach is a point embedding network that generates a vectorial representation of the 3D points such that it reflects the grouping hierarchy and tag data. The main challenge is that the data is noisy and highly variable. To this end, we present a tree-aware metriclearning approach and demonstrate that such learned embeddings offer excellent transfer to semantic segmentation tasks, especially when training data is limited. Our approach reduces the relative error by 10.2% with 8 training examples, by 11.72% with 120 training examples on the ShapeNet semantic segmentation benchmark, in comparison to the network trained from scratch. By utilizing tag data the relative error is reduced by 12.8% with 8 training examples, in comparison to the network trained from scratch. These improvements come at no additional labeling cost as the meta data is freely available.

PriFit: Learning to Fit Primitives Improves Few Shot Point Cloud Segmentation

“…Several works have proposed learning point representation using noisy labels and semantic tags available from various shape repositories. Sharma et al [42] learn point representations using noisy part hierarchies and designer-labeled semantic tags for few-shot semantic segmentation. Muralikrishnan et al [34] design a U-Net to learn point representations that predicts userprescribed shape-level tags by first predicting intermediate semantic segmentation.…”

Label-Efficient Learning on Point Clouds using Approximate Convex Decompositions

“…Label-efficient representation learning on point clouds. Several recent approaches [8,20,35,46,74] have been proposed to alleviate expensive labeling of shapes. Muralikrishnan et al [35] learn per-point representation by training the network to predict shape-level tags.…”

“…If we further add a reconstruction term, our method achieves state-of-the-art performance in unsuperivsed shape classification. Finally, Sharma et al [46] proposed learning point embedding by utilizing noisy part labels and semantic tags available freely on a 3D warehouse dataset. The model learnt in this way is used for a few-shot semantic segmentation task.…”

“…The performance of current deep neural network models on tasks such as classification and semantic segmentation of point cloud data is limited by the amount of high quality labeled data available for training the networks. Since in many situations collecting high quality annotations on point cloud data is time consuming and incurs a high cost, there has been increasing efforts in circumventing this problem by training the neural networks on noisy or weakly labeled datasets [46], or training in completely unsupervised ways [8,14,20,66,67].…”

A Progressive Conditional Generative Adversarial Network for Generating Dense and Colored 3D Point Clouds

“…Generating synthetic 3D point cloud data is an open area of research with the intention of facilitating the learning of non-Euclidean point representations. In three dimensions, synthetic data may take the form of meshes, voxels, or raw point clouds in order to learn a representation that aids the solution of computer vision tasks such as classification [34,45,29,59,9], segmentation [34,45,35,55,19,56,39,51], and reconstruction [50,46,48,38,57,42]. Currently, researchers make use of point clouds sampled from the mesh of manually designed objects as synthetic data for training deep learning models [34,45,40,7].…”