Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

Chen, Kevin; Choy, Christopher; Savva, Manolis; Chang, Angel X.; Savarese, Silvio

doi:10.1007/978-3-030-20893-6_7

Cited by 99 publications

(116 citation statements)

References 39 publications

(92 reference statements)

Supporting

Mentioning

106

Contrasting

Order By: Relevance

“…Hu et al [2] use a similar approach albeit using deep neural networks to avoid parsing and feature construction by hand. Chen et al [5] learn joint embeddings of language descriptions and colored 3D objects. These approaches are trained on image data collected from internet sources and may differ from data observed on a robot's camera.…”

Section: Background and Related Workmentioning

confidence: 99%

“…One of the critical challenges in robotics is scaling techniques to work in a wide array of environments. While the results presented in this paper constitute an important step towards grounding objects to natural language, existing work in this area still generally only applies to a small number, 1 -4, of object classes [5], [25]. Moving forward, more expansive datasets, of both natural language object descriptions and 3D shapes, are necessary to enable larger systems to be learned.…”

Section: Future Directionsmentioning

confidence: 99%

See 1 more Smart Citation

Grounding Language Attributes to Objects using Bayesian Eigenobjects

Cohen

Burchfiel

Nguyen

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

We develop a system to disambiguate object instances within the same class based on simple physical descriptions. The system takes as input a natural language phrase and a depth image containing a segmented object and predicts how similar the observed object is to the object described by the phrase. Our system is designed to learn from only a small amount of human-labeled language data and generalize to viewpoints not represented in the language-annotated depth image training set. By decoupling 3D shape representation from language representation, this method is able to ground language to novel objects using a small amount of languageannotated depth-data and a larger corpus of unlabeled 3D object meshes, even when these objects are partially observed from unusual viewpoints. Our system is able to disambiguate between novel objects, observed via depth images, based on natural language descriptions. Our method also enables viewpoint transfer; trained on human-annotated data on a small set of depth images captured from frontal viewpoints, our system successfully predicted object attributes from rear views despite having no such depth images in its training set. Finally, we demonstrate our approach on a Baxter robot, enabling it to pick specific objects based on human-provided natural language descriptions.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Future Directionsmentioning

confidence: 99%

Grounding Language Attributes to Objects using Bayesian Eigenobjects

Cohen

Burchfiel

Nguyen

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…Then a long-term recurrent convolutional network (LRCN) was employed to refine the generated results to obtain more complete 3D models in higher resolution. Chen et al [29] proposed text2shape system which combined 3D generation with natural language processing. The network encoded the text, then regarded the results as a condition, and utilized WGAN to decode it into a 3D model related to input text.…”

Section: D Gansmentioning

confidence: 99%

3D Model Generation and Reconstruction Using Conditional Generative Adversarial Network

Li¹,

Zheng²,

Wu³

et al. 2019

IJCIS

View full text Add to dashboard Cite

A B S T R A C TGenerative adversarial network (GANs) has significant progress in 3D model generation and reconstruction recently years. GANs can generate 3D models by sampling from uniform noise distribution. But they generate randomly and are often not easy to control. To address this problem, we add the class information to both generator and discriminator and construct a new network named 3D conditional GAN. Moreover, to better guide generator to reconstruct 3D model from a single image in high quality, we propose a new 3D model reconstruction network by integrating a classifier into the traditional system. Experimental results on ModelNet10 dataset show that our method can effectively generate realistic 3D models corresponding to the given class labels. And the qualities of 3D model reconstruction have been improved considerably by using proposed method in IKEA dataset.

show abstract

“…The priors are not specific to individual shapes. Chen et al [11] gather natural language descriptions for 3D shapes that sometimes include material labels ("This is a brown wooden chair"), but there is no fine-grained region labeling that can be used for training. Yang et al [48] propose a data-driven algorithm to reshape shape components to a target fabrication material.…”

Section: Previous Workmentioning

confidence: 99%

Learning Material-Aware Local Descriptors for 3D Shapes

Lin

Averkiou

Kalogerakis

et al. 2018

2018 International Conference on 3D Vision (3DV)

View full text Add to dashboard Cite

Material understanding is critical for design, geometric modeling, and analysis of functional objects. We enable material-aware 3D shape analysis by employing a projective convolutional neural network architecture to learn materialaware descriptors from view-based representations of 3D points for point-wise material classification or materialaware retrieval. Unfortunately, only a small fraction of shapes in 3D repositories are labeled with physical materials, posing a challenge for learning methods. To address this challenge, we crowdsource a dataset of 3080 3D shapes with part-wise material labels. We focus on furniture models which exhibit interesting structure and material variability. In addition, we also contribute a high-quality expertlabeled benchmark of 115 shapes from Herman-Miller and IKEA for evaluation. We further apply a mesh-aware conditional random field, which incorporates rotational and reflective symmetries, to smooth our local material predictions across neighboring surface patches. We demonstrate the effectiveness of our learned descriptors for automatic texturing, material-aware retrieval, and physical simulation. The dataset and code will be publicly available.

show abstract

Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings

Cited by 99 publications

References 39 publications

Grounding Language Attributes to Objects using Bayesian Eigenobjects

Grounding Language Attributes to Objects using Bayesian Eigenobjects

3D Model Generation and Reconstruction Using Conditional Generative Adversarial Network

Learning Material-Aware Local Descriptors for 3D Shapes

Contact Info

Product

Resources

About