Text-To-4D Dynamic Scene Generation

Singer, Uriel; Sheynin, Shelly; Polyak, Adam; Ashual, Oron; Makarov, Iurii; Kokkinos, Filippos; Goyal, Naman; Vedaldi, Andrea; Parikh, Devi; Johnson, Justin C.; Taigman, Yaniv

doi:10.48550/arxiv.2301.11280

Cited by 2 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our method focuses on 4D NeRF generation from text, which is more challenging than those text/image-based 3D generation methods mentioned above. The pioneering work, MAV3D [25] introduces video-based SDS loss and adopts dynamic NeRF representation, HexPlane [26]. It divides the generation process into three stages: static, dynamic, and super-resolution, but the generation quality can be further improved.…”

Section: Text/video-guided 4d Generationmentioning

confidence: 99%

“…The dynamic modeling method will affect the final dynamic result. The existing dynamic generation methods either adopt a deformation field network [106], which can model motion continuity well but is limited by the topological change, or they introduce additional dynamic feature inputs through temporal feature grids [25], [107], which can model diverse motions but lacks continuity guarantee. Therefore, we propose to exploit the combination of a deformation network and a topology network to ensure motion continuity while breaking the limitation of topology and enriching motion types.…”

Section: D Representationmentioning

confidence: 99%

“…The overall optimization loss L dynamic in the dynamic stage is: (6) where λ * are the weights for corresponding losses. For the specific definition of all loss terms and the default values of [25] and 4d-fy [107]. Our method has significant advantages over MAV3D.…”

Section: D Generation With Video Priormentioning

confidence: 99%

“…However, existing data does not support training a large model for 4D generation, so current methods use SDS to distill from a 2D generative model. The pioneering work, MAV3D [25], adopts HexPlane [26] as the dynamic representation and divides the generation into three stages: static, dynamic, and super-resolution with a combination of image SDS loss and video SDS loss. The latter one distills 4D dynamic information from the text-tovideo diffusion model.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Perovskite-loaded plasmonic gold nanorod composites enhanced solar cell performance

Yuan

Liu

et al. 2023

Adv Compos Hybrid Mater

View full text Add to dashboard Cite

Due to the fascinating generative performance of text-to-image diffusion models, growing text-to-3D generation works explore distilling the 2D generative priors into 3D, using the score distillation sampling (SDS) loss, to bypass the data scarcity problem.The existing text-to-3D methods have achieved promising results in realism and 3D consistency, but text-to-4D generation still faces challenges, including lack of realism and insufficient dynamic motions. In this paper, we propose a novel method for text-to-4D generation, which ensures the dynamic amplitude and authenticity through direct supervision provided by a video prior. Specifically, we adopt a text-to-video diffusion model to generate a reference video and divide 4D generation into two stages: static generation and dynamic generation. The static 3D generation is achieved under the guidance of the input text and the first frame of the reference video, while in the dynamic generation stage, we introduce a customized SDS loss to ensure multi-view consistency, a video-based SDS loss to improve temporal consistency, and most importantly, direct priors from the reference video to ensure the quality of geometry and texture. Moreover, we design a prior-switching training strategy to avoid conflicts between different priors and fully leverage the benefits of each prior. In addition, to enrich the generated motion, we further introduce a dynamic modeling representation composed of a deformation network and a topology network, which ensures dynamic continuity while modeling topological changes. Our method not only supports text-to-4D generation but also enables 4D generation from monocular videos. The comparison experiments demonstrate the superiority of our method compared to existing methods.

show abstract

Section: Text/video-guided 4d Generationmentioning

confidence: 99%

Section: D Representationmentioning

confidence: 99%

Section: D Generation With Video Priormentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Perovskite-loaded plasmonic gold nanorod composites enhanced solar cell performance

Yuan

Liu

et al. 2023

Adv Compos Hybrid Mater

View full text Add to dashboard Cite

show abstract

“…With widespread use, the applications for NeRF become ever more complex, increasing the demand for faster training. Examples for complex applications are Text-to-NeRF approaches [34]- [36] or NeRF-based Text-to-Video approaches [37]. Other applications include e.g.…”

Section: Applicationsmentioning

confidence: 99%