Towards Implicit Text-Guided 3D Shape Generation

Liu, Zhengzhe; Wang, Yi; Qi, Xiaojuan

doi:10.1109/cvpr52688.2022.01737

Cited by 43 publications

(45 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Relying on a pre-trained CLIP model, CLIPMesh directly performs optimization on mesh parameters to generate shape and texture that enables to render images matching the input text prompt. Instead of using CLIP, Liu et al [202] adopt a BERT-based text encoder and achieve text-guided 3D shape generation with high-fidelity.…”

Section: Other Methodsmentioning

confidence: 99%

Expired

Zhan¹

2022

Preprint

View full text Add to dashboard Cite

As information exists in various modalities in real world, effective interaction and fusion among multimodal information plays a key role for the creation and perception of multimodal data in computer vision and deep learning research. With superb power in modelling the interaction among multimodal information, multimodal image synthesis and editing have become a hot research topic in recent years. Different from traditional visual guidance which provides explicit clues, multimodal guidance offers intuitive and flexible means in image synthesis and editing. On the other hand, this field is also facing several challenges in alignment of features with inherent modality gaps, synthesis of high-resolution images, faithful evaluation metrics, etc. In this survey, we comprehensively contextualize the advance of the recent multimodal image synthesis \& editing and formulate taxonomies according to data modality and model architectures. We start with an introduction to different types of guidance modalities in image synthesis and editing. We then describe multimodal image synthesis and editing approaches extensively with detailed frameworks including Generative Adversarial Networks (GANs), GAN Inversion, Transformers, and other methods such as NeRF and Diffusion models. This is followed by a comprehensive description of benchmark datasets and corresponding evaluation metrics as widely adopted in multimodal image synthesis and editing, as well as detailed comparisons of different synthesis methods with analysis of respective advantages and limitations. Finally, we provide insights into the current research challenges and possible future research directions. We hope this survey could lay a sound and valuable foundation for future development of multimodal image synthesis and editing. A project associated with this survey is available at \href{https://github.com/fnzhan/MISE}{https://github.com/fnzhan/MISE}.

show abstract

Section: Other Methodsmentioning

confidence: 99%

Expired

Zhan¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…There are mainly two categories of methods, i.e., fully-supervised and optimizationbased methods. The fully-supervised method [6,8,25] uses ground truth text and the paired 3D objects with explicit 3D representations as training data. Specifically, CLIP-Forge [43] uses a two-stage training scheme, which consists of shape autoencoder training, and conditional normalizing flow training.…”

Section: Text-guided 3d Shape Generationmentioning

confidence: 99%

“…In Fig. 6, we first compare with the method TITG3SG [25], which is an encoder-decoder-based framework. They only experiment with object classes of Chair and Table . Here we conduct the comparison qualitatively.…”

Section: Qualitative Comparisonmentioning

confidence: 99%

See 1 more Smart Citation

TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision

Wei¹,

Wang²,

Feng³

et al. 2023

Preprint

View full text Add to dashboard Cite

a red hatchback car" "a gray armless chair" "a wooden office table" "a black motorbike" Figure 1. Our proposed model is capable of generating detailed textured shapes based on given text prompts. We demonstrate the effectiveness of our approach by showcasing generated results for four distinct object classes: Car, Chair, Table, and Motorbike. Both the textured meshes and geometries are presented, with visualizations rendered using ChimeraX [10].

show abstract

“…Colorization can be formulated to this task and handled by Generative Adversarial Networks [11] (GAN) based approaches [19,41,35,30,44]. They employ an adversarial loss that learns to discriminate between real and generated images, and then minimize this loss by updating the generator to make the produced results look realistic [57,28,31,50,36,51,45,46,42].…”

Section: Gan-based Image-to-image Translationmentioning

confidence: 99%