Learning Local Shape Descriptors from Part Correspondences with Multiview Convolutional Networks

Abstract

partial scan to shape matching shape segmentation keypoint matching affordance prediction ("palm") i view based convolutional network .. . .. . ... point descriptor Fig. 1. We present a view-based convolutional network that produces local, point-based shape descriptors. The network is trained such that geometrically and semantically similar points across different 3D shapes are embedded close to each other in descriptor space (left). Our produced descriptors are quite generic -they can be used in a variety of shape analysis applications, including dense matching, prediction of human affordance regions, partial scan-to-shape matching, and shape segmentation (right).We present a new local descriptor for 3D shapes, directly applicable to a wide range of shape analysis problems such as point correspondences, semantic segmentation, affordance prediction, and shape-to-scan matching. The descriptor is produced by a convolutional network that is trained to embed geometrically and semantically similar points close to one another in descriptor space. The network processes surface neighborhoods around points on a shape that are captured at multiple scales by a succession of progressively zoomed out views, taken from carefully selected camera positions. We leverage two extremely large sources of data to train our network. First, since our network processes rendered views in the form of 2D images, we repurpose architectures pre-trained on massive image datasets. Second, we automatically generate a synthetic dense point correspondence dataset by non-rigid alignment of corresponding shape parts in a large collection of segmented 3D models. As a result of these design choices, our network effectively encodes multi-scale local context and fine-grained surface detail. Our network can be trained to produce either category-specific descriptors or more generic descriptors by learning from multiple shape categories. Once trained, at test time, the network extracts local descriptors for shapes without requiring any part segmentation as input. Our method can produce effective local descriptors even for shapes whose category is unknown or different from the ones used while training. We demonstrate through several experiments that our learned local descriptors are more discriminative © 2017 Association for Computing Machinery. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record is published in ACM Transactions on Graphics.compared to state of the art alternatives, and are effective in a variety of shape analysis applications.

SiamesePointNet: A Siamese Point Network Architecture for Learning 3D Shape Descriptor

“…The similar work is given by Huang et al . [HKC*18]. This method produces a single, compact representation of a 3D shape by aggregating information across multiple views by comparing the learned descriptors between correspondence points via a siamese network architecture.…”

“…[ZSN*17] and Huang et al . [HKC*18], we also use a siamese network architecture to learn point descriptors from correspondence shapes.…”

“…A large correspondence dataset [HKC*18] is used for training shape correspondence. These correspondence pairs are generated via a part‐based registration method and all the shapes are from ShapeNetCore dataset [XKH*16].…”

“…Then Huang et al . [HKC*18] generate a synthetic dense point correspondence dataset by non‐rigid alignment of corresponding shape parts in a large collection of segmented ShapeNetCore dataset [YKC*16]. This dataset contains rich correspondence training samples of multi‐category 3D shapes.…”

“…Inspired by the recent success of neural networks using in 3D points correspondence, such as 3DMatch [ZSN*17] and multi‐view correspondence [HKC*18], we formulate a data‐driven method to learn geometric descriptors on each point for establishing correspondences between 3D shapes. We use a Siamese network to learn point‐wise semantic descriptors considering 3D shapes spatial distribution and deformation between pairs of shapes.…”