RetrievalFuse: Neural 3D Scene Reconstruction with a Database

Siddiqui, Yawar; Thies, Justus; Ma, Fangchang; Qi, Shihua; Nießner, Matthias; Dai, Angela

doi:10.1109/iccv48922.2021.01234

Cited by 25 publications

(16 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, although it can cope with a substantial variation of point density, it cannot complete shapes when large parts are missing. Apart from a few methods like [26,27,25,98] 8.58 0.936 0.9787 Neural Splines [118] 5.99 0.982 0.9958 LIG [49] 8.69 0.975 0.9773 POCO (ours) 5.27 0.992 0.9987 1000 SPR10 [51] 7.29 0.967 0.9957 LIG [49] 8.40 0.978 0.9750 POCO (ours) 5.34 0.993 0.9987 Oracle 5.02 0.995 0.9998 [118] uses a grid size of 1024, 10k Nyström samples, 8×8×8 chunks. Numbers differ from [49] as we had to regenerate the unavailable watertight meshes.…”

Section: Discussion and Limitationsmentioning

confidence: 99%

“…Besides, we use a learned interpolation rather than the usual tri-linear interpolation [18,49,87,19,62,105] or the inverse-distance distance weighting [90]. Although different in nature, learning has also been used in [98] to blend retrieved chunks.…”

Section: Convolutions For Implicit Representationsmentioning

confidence: 99%

“…RetrievalFuse [98] splits a scene along a regular grid and encodes each 3D chunk as a latent vector via convolutional layers. But rather than using them for decoding, it retrieves similar chunks from the training set and combines their distance field to create a surface, enhancing the completion capability.…”

Section: Convolutions For Implicit Representationsmentioning

confidence: 99%

See 2 more Smart Citations

POCO: Point Convolution for Surface Reconstruction

Boulch¹,

Marlet²

2022

Preprint

View full text Add to dashboard Cite

Implicit neural networks have been successfully used for surface reconstruction from point clouds. However, many of them face scalability issues as they encode the isosurface function of a whole object or scene into a single latent vector. To overcome this limitation, a few approaches infer latent vectors on a coarse regular 3D grid or on 3D patches, and interpolate them to answer occupancy queries. In doing so, they loose the direct connection with the input points sampled on the surface of objects, and they attach information uniformly in space rather than where it matters the most, i.e., near the surface. Besides, relying on fixed patch sizes may require discretization tuning. To address these issues, we propose to use point cloud convolutions and compute latent vectors at each input point. We then perform a learning-based interpolation on nearest neighbors using inferred weights. Experiments on both object and scene datasets show that our approach significantly outperforms other methods on most classical metrics, producing finer details and better reconstructing thinner volumes. The code is available at https://github.com/ valeoai/POCO.

show abstract

Section: Discussion and Limitationsmentioning

confidence: 99%

Section: Convolutions For Implicit Representationsmentioning

confidence: 99%

See 1 more Smart Citation

POCO: Point Convolution for Surface Reconstruction

Boulch¹,

Marlet²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…RetrieveGAN [44] uses a differentiable retrieval module for image generation from scene description. Moreover, RetrievalFuse [39] proposed a neural 3D scene reconstruction based on a retrieval system. The authors showed that the retrieval method enables more accurate 3D reconstruction.…”

Section: Retrieval For Generationmentioning

confidence: 99%

KNN-Diffusion: Image Generation via Large-Scale Retrieval

Ashual¹,

Sheynin²,

Polyak³

et al. 2022

Preprint

View full text Add to dashboard Cite

While the availability of massive Text-Image datasets is shown to be extremely useful in training large-scale generative models (e.g. DDPMs, Transformers), their output typically depends on the quality of both the input text, as well as the training dataset. In this work, we show how largescale retrieval methods, in particular efficient K-Nearest-Neighbors (KNN) search, can be used in order to train a model to adapt to new samples. Learning to adapt enables several new capabilities. Sifting through billions of records at inference time is extremely efficient and can alleviate the need to train or memorize an adequately large generative model. Additionally, fine-tuning trained models to new samples can be achieved by simply adding them to the table. Rare concepts, even without any presence in the training set, can be then leveraged during test time without any modification to the generative model. Our diffusion-based model trains on images only, by leveraging a joint Text-Image multi-modal metric. Compared to baseline methods, our generations achieve state of the art results both in human evaluations as well as with perceptual scores when tested on a public multimodal dataset of natural images, as well as on a collected dataset of 400 million Stickers.

show abstract

“…These retrieval-augmented models with external memory turn purely parametric deep learning models into semi-parametric ones. Early attempts [36,50,57,63] in retrieval-augmented visual models do not use an external memory and exploit the training data itself for retrieval. In image synthesis, IC-GAN [5] utilizes the neighborhood of training images to train a GAN and generates samples by conditioning on single instances from the training data.…”

Section: Related Workmentioning

confidence: 99%

Retrieval-Augmented Diffusion Models

Blattmann¹,

Rombach²,

Oktay³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generative image synthesis with diffusion models has recently achieved excellent visual quality in several tasks such as text-based or class-conditional image synthesis. Much of this success is due to a dramatic increase in the computational capacity invested in training these models. This work presents an alternative approach: inspired by its successful application in natural language processing, we propose to complement the diffusion model with a retrieval-based approach and to introduce an explicit memory in the form of an external database. During training, our diffusion model is trained with similar visual features retrieved via CLIP and from the neighborhood of each training instance. By leveraging CLIP's joint image-text embedding space, our model achieves highly competitive performance on tasks for which it has not been explicitly trained, such as class-conditional or text-image synthesis, and can be conditioned on both text and image embeddings. Moreover, we can apply our approach to unconditional generation, where it achieves state-of-the-art performance. Our approach incurs low computational and memory overheads and is easy to implement. We discuss its relationship to concurrent work and will publish code and pretrained models soon.

show abstract

RetrievalFuse: Neural 3D Scene Reconstruction with a Database

Cited by 25 publications

References 25 publications

POCO: Point Convolution for Surface Reconstruction

POCO: Point Convolution for Surface Reconstruction

KNN-Diffusion: Image Generation via Large-Scale Retrieval

Retrieval-Augmented Diffusion Models

Contact Info

Product

Resources

About