2D GANs Meet Unsupervised Single-View 3D Reconstruction

Liu, Feng; Liu, Xiaoming

doi:10.1007/978-3-031-19769-7_29

Cited by 10 publications

(5 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other interesting works include: SIREN [138] and Fourier Feature Networks [142] for improving the representation ability of neural implicit by applying sine activation functions and Fourier feature mapping, respectively; InstantNGP [104] for greatly improving the training speed by applying multi-resolution Hash encoding; [89] for introducing a method to reconstruct 3D shapes from single-view 2D images with the help of pseudo multi-view images generated by StyleGAN [70]; MeshSDF [121] for designing differentiable iso-surface extraction algorithm (Marching Cubes) on neural implicit; SAL (Sign Agnostic Learning) [121] for proposing a method to learn neural implicit from unoriented point clouds; Neu-ralUDF [27] and MeshUDF [50] for proposing Neural unsigned distance fields and how to extract explicit meshes from them, respectively; GIFS [179] for proposing a neural field to represent general shapes including non-watertight shapes and shapes with multi-layer surfaces, by predicting whether two points are separated by any surface, instead of the inside-outside of each point.…”

Section: Neural Implicitmentioning

confidence: 99%

See 1 more Smart Citation

A Review of Deep Learning-Powered Mesh Reconstruction Methods

Feng¹

2023

Preprint

View full text Add to dashboard Cite

With the recent advances in hardware and rendering techniques, 3D models have emerged everywhere in our life. Yet creating 3D shapes is arduous and requires significant professional knowledge. Meanwhile, Deep learning has enabled high-quality 3D shape reconstruction from various sources, making it a viable approach to acquiring 3D shapes with minimal effort. Importantly, to be used in common 3D applications, the reconstructed shapes need to be represented as polygonal meshes, which is a challenge for neural networks due to the irregularity of mesh tessellations.In this survey, we provide a comprehensive review of mesh reconstruction methods that are powered by machine learning. We first describe various representations for 3D shapes in the deep learning context. Then we review the development of 3D mesh reconstruction methods from voxels, point clouds, single images, and multi-view images. Finally, we identify several challenges in this field and propose potential future directions.

show abstract

Section: Neural Implicitmentioning

confidence: 99%

“…Single-View Reconstruction by Cross-Instance Consistency" (UNICORN) [103] and Table 1 in "2D GANs Meet Unsupervised Single-view 3D Reconstruction" (GANSVR) [89] are great summaries of recent works on this topic. We recommend interested readers to take a look.…”

Section: With 2d Supervisionmentioning

confidence: 99%

A Review of Deep Learning-Powered Mesh Reconstruction Methods

Feng¹

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Another popular method recently is to learn a neural network which approximates the implicit function [16][17][18][19][20][21]. Besides, methods based on generative adversarial network (GAN) have been applied to 3D shape reconstruction or style reconstruction after making significant progress in 2D generation [22][23][24][25][26]. These methods have accomplished remarkable results, but they lack interpretability or cannot provide a detailed step-by-step reconstruction process.…”

Section: Related Workmentioning

confidence: 99%

3D reconstruction based on hierarchical reinforcement learning with transferability

Fan

et al. 2023

ICA

View full text Add to dashboard Cite

3D reconstruction is extremely important in CAD (computer-aided design)/CAE (computer-aided Engineering)/CAM (computer-aided manufacturing). For interpretability, reinforcement learning (RL) is used to reconstruct 3D shapes from images by a series of editing actions. However, typical applications of RL for 3D reconstruction face problems. The search space will increase exponentially with the action space due to the curse of dimensionality, which leads to low performance, especially for complex action spaces in 3D reconstruction. Additionally, most works involve training a specific agent for each shape class without learning related experiences from others. Therefore, we present a hierarchical RL approach with transferability to reconstruct 3D shapes (HRLT3D). First, actions are grouped into macro actions that can be chosen by the top-agent. Second, the task is accordingly decomposed into hierarchically simplified sub-tasks solved by sub-agents. Different from classical hierarchical RL (HRL), we propose a sub-agent based on augmented state space (ASS-Sub-Agent) to replace a set of sub-agents, which can speed up the training process due to shared learning and having fewer parameters. Furthermore, the ASS-Sub-Agent is more easily transferred to data of other classes due to the augmented diverse states and the simplified tasks. The experimental results on typical public dataset show that the proposed HRLT3D performs overwhelmingly better than recent baselines. More impressingly, the experiments also demonstrate the extreme transferability of our approach among data of different classes.

show abstract

“…It introduced a mapping network, which mapped the sampled noise into another latent space, which is more disentangled and semantically coherent, as demonstrated by its downstream usage for image editing and manipulation [1,33,34,43,44]. Further, StyleGAN has been extended to get novel views from images [27,29,45], thus making it possible to get 3D information from it. These downstream advances are possible due to the impressive performance of StyleGANs on class-specific datasets (such as faces).…”

Section: Related Workmentioning

confidence: 99%

Improving GANs for Long-Tailed Data Through Group Spectral Regularization

Rangwani

Jaswani

Karmali

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Deep long-tailed learning aims to train useful deep networks on practical, real-world imbalanced distributions, wherein most labels of the tail classes are associated with a few samples. There has been a large body of work to train discriminative models for visual recognition on long-tailed distribution. In contrast, we aim to train conditional Generative Adversarial Networks, a class of image generation models on long-tailed distributions. We find that similar to recognition, stateof-the-art methods for image generation also suffer from performance degradation on tail classes. The performance degradation is mainly due to class-specific mode collapse for tail classes, which we observe to be correlated with the spectral explosion of the conditioning parameter matrix. We propose a novel group Spectral Regularizer (gSR) that prevents the spectral explosion alleviating mode collapse, which results in diverse and plausible image generation even for tail classes. We find that gSR effectively combines with existing augmentation and regularization techniques, leading to state-of-the-art image generation performance on long-tailed data. Extensive experiments demonstrate the efficacy of our regularizer on long-tailed datasets with different degrees of imbalance. Project Page: https://sites.google.com/view/gsr-eccv22.

show abstract

2D GANs Meet Unsupervised Single-View 3D Reconstruction

Cited by 10 publications

References 46 publications

A Review of Deep Learning-Powered Mesh Reconstruction Methods

A Review of Deep Learning-Powered Mesh Reconstruction Methods

3D reconstruction based on hierarchical reinforcement learning with transferability

Improving GANs for Long-Tailed Data Through Group Spectral Regularization

Contact Info

Product

Resources

About