SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis

Wang, Guangcong; Chen, Zhaoxi; Loy, Chen Change; Liu, Ziwei

doi:10.48550/arxiv.2303.16196

Cited by 4 publications

(4 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lastly, our method attains state-of-the-art results across various tasks. Future work will extend the application of our DNA method to NLP, 3D architectures [98], [99], [100], and generative models [101], [102], [103].…”

Section: Discussionmentioning

confidence: 99%

DNA Family: Boosting Weight-Sharing NAS With Block-Wise Supervisions

Wang,

Li,

Yuan

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Neural Architecture Search (NAS), aiming at automatically designing neural architectures by machines, has been considered a key step toward automatic machine learning. One notable NAS branch is the weight-sharing NAS, which significantly improves search efficiency and allows NAS algorithms to run on ordinary computers. Despite receiving high expectations, this category of methods suffers from low search effectiveness. By employing a generalization boundedness tool, we demonstrate that the devil behind this drawback is the untrustworthy architecture rating with the oversized search space of the possible architectures. Addressing this problem, we modularize a large search space into blocks with small search spaces and develop a family of models with the distilling neural architecture (DNA) techniques. These proposed models, namely a DNA family, are capable of resolving multiple dilemmas of the weight-sharing NAS, such as scalability, efficiency, and multi-modal compatibility. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub-search space using heuristic algorithms. Moreover, under a certain computational complexity constraint, our method can seek architectures with different depths and widths. Extensive experimental evaluations show that our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively. Additionally, we provide in-depth empirical analysis and insights into neural architecture ratings. Codes available: https://github.com/changlin31/DNA.

show abstract

Section: Discussionmentioning

confidence: 99%

DNA Family: Boosting Weight-Sharing NAS With Block-Wise Supervisions

Wang,

Li,

Yuan

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…Additionally, 40 test images without the object are provided for quantitative evaluations. When assessing depth quality, we adhere to a standard scheme [31] of training a robust NeRF model exclusively on test views to produce pseudo-groundtruth depth maps. It is also important to note that both RGB and depth images have been resized to 1008 × 567, aligning with the dimensions used in [17].…”

Section: Methodsmentioning

confidence: 99%

Fast virtual view synthesis for an 8K 3D light-field display based on cutoff-NeRF and 3D voxel rendering

chen¹,

Yan²,

Sang³

et al. 2022

Opt. Express

View full text Add to dashboard Cite

Three-dimensional (3D) light-field displays can provide an immersive visual experience, which has attracted significant attention. However, the generating of high-quality 3D light-field content in the real world is still a challenge because it is difficult to capture dense high-resolution viewpoints of the real world with the camera array. Novel view synthesis based on CNN can generate dense high-resolution viewpoints from sparse inputs but suffer from high-computational resource consumption, low rendering speed, and limited camera baseline. Here, a two-stage virtual view synthesis method based on cutoff-NeRF and 3D voxel rendering is presented, which can fast synthesize dense novel views with smooth parallax and 3D images with a resolution of 7680 × 4320 for the 3D light-field display. In the first stage, an image-based cutoff-NeRF is proposed to implicitly represent the distribution of scene content and improve the quality of the virtual view. In the second stage, a 3D voxel-based image rendering and coding algorithm is presented, which quantify the scene content distribution learned by cutoff-NeRF to render high-resolution virtual views fast and output high-resolution 3D images. Among them, a coarse-to-fine 3D voxel rendering method is proposed to improve the accuracy of voxel representation effectively. Furthermore, a 3D voxel-based off-axis pixel encoding method is proposed to speed up 3D image generation. Finally, a sparse views dataset is built by ourselves to analyze the effectiveness of the proposed method. Experimental results demonstrate the method’s effectiveness, which can fast synthesize novel views and 3D images with high resolution in real 3D scenes and physical simulation environments. PSNR of the virtual view is about 29.75 dB, SSIM is about 0.88, and the synthetic 8K 3D image time is about 14.41s. We believe that our fast high-resolution virtual viewpoint synthesis method can effectively improve the application of 3D light field display.

show abstract

“…To address this issue, Wei et al [45] introduced additional conditions on the confidence of depth information to limit the sampling range, reducing the impact of erroneous depth information. Wang et al [46] propose a simple yet effective constraint, a local depth ranking method, on NeRFs such that the expected depth ranking of the NeRF is consistent with that of the coarse depth maps in local patches. And they further propose a spatial continuity constraint to encourage the consistency of the expected depth continuity of NeRF with coarse depth maps.…”

Section: Nerf With Depthmentioning

confidence: 99%

Dip-NeRF: Depth-Based Anti-Aliased Neural Radiance Fields

Qin,

Xiao,

2024

Electronics

View full text Add to dashboard Cite

Neural radiation field (NeRF)-based novel view synthesis methods are gaining popularity for their ability to generate detailed and realistic images. However, most NeRF-based methods only use images to learn scene representations, ignoring the importance of depth information. The Zip-NeRF method has achieved impressive results in unbounded scenes by combining anti-aliasing techniques and mesh representations. However, the method requires a large number of input images and may perform poorly in complex scenes. Our method incorporates the advantages of Zip-NeRF and incorporates depth information to reduce the number of required images and solve the scale-free problem in borderless scenes. Experimental results show that our method effectively reduces the training time.And we can generate high-quality images and fine point cloud models using few images, even in complex scenes with numerous occlusions.

show abstract

SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis

Cited by 4 publications

References 46 publications

DNA Family: Boosting Weight-Sharing NAS With Block-Wise Supervisions

DNA Family: Boosting Weight-Sharing NAS With Block-Wise Supervisions

Fast virtual view synthesis for an 8K 3D light-field display based on cutoff-NeRF and 3D voxel rendering

Dip-NeRF: Depth-Based Anti-Aliased Neural Radiance Fields

Contact Info

Product

Resources

About