Fine-Grained VR Sketching: Dataset and Insights

Abstract

We present the first fine-grained dataset of 1,497 3D VR sketch and 3D shape pairs of a chair category with large shapes diversity. Our dataset supports the recent trend in the sketch community on fine-grained data analysis, and extends it to an actively developing 3D domain. We argue for the most convenient sketching scenario where the sketch consists of sparse lines and does not require any sketching skills, prior training or time-consuming accurate drawing. We then, for the first time, study the scenario of fine-grained 3D VR sketch to 3D shape retrieval, as a novel VR sketching application and a proving ground to drive out generic insights to inform future research. By experimenting with carefully selected combinations of design factors on this new problem, we draw important conclusions to help follow-on work. We hope our dataset will enable other novel applications, especially those that require a fine-grained angle such as fine-grained 3D shape reconstruction. The dataset is available at tinyurl.com/ VRSketch3DV21.

Towards 3D VR-Sketch to 3D Shape Retrieval

“…Following [23], we represent 3D shapes and 3D sketches as point clouds, and train the model via a Siamese training with a triplet loss [35,28,36,11]. As a 3D sketch and shape encoder we exploit PointNet++ [26], where the same set of weights is used to encode both modalities.…”

“…We denote the feature embedding of a sketch S as s ∈ R 512 (source) and a shape T as t ∈ R 512 (target). For a given batch of N sketch-shape pairs, in [23] the triplet loss is defined as follows:…”

“…However, despite recent attempts to address the domain gap between 3D shapes and 2D sketches [24,20], instance-level retrieval of 3D shapes from 2D sketches remains challenging at large. On the contrary, recent works [22,23] demonstrate the advantage of 3D VR sketches for 3D shapes retrieval. Despite encouraging results, these models return many irrelevant shapes among the top retrieval results (Fig.…”

“…In their case, the VR sketch comprises a set of dense ribbon-like strokes. Luo et al [23] studied FG-3D-VR-SBSR from sparse quick sketches and introduced the first dataset comprising 1,497 3D VR sketch and 3D shape pairs of chairs. They performed benchmarking on common 3D shape retrieval methods and demonstrated encouraging results.…”

“…For training and testing we exploit the dataset of 3D VR sketches by Luo et al [23]. This dataset consists of 1,005 unique 3D sketch and shape pairs from the ShapeNet chair category, created by 50 participants.…”