Specular-to-Diffuse Translation for Multi-view Reconstruction

Abstract

Most multi-view 3D reconstruction algorithms, especially when shapefrom-shading cues are used, assume that object appearance is predominantly diffuse. To alleviate this restriction, we introduce S2Dnet, a generative adversarial network for transferring multiple views of objects with specular reflection into diffuse ones, so that multi-view reconstruction methods can be applied more effectively. Our network extends unsupervised image-to-image translation to multiview "specular to diffuse" translation. To preserve object appearance across multiple views, we introduce a Multi-View Coherence loss (MVC) that evaluates the similarity and faithfulness of local patches after the view-transformation. Our MVC loss ensures that the similarity of local correspondences among multi-view images is preserved under the image-to-image translation. As a result, our network yields significantly better results than several single-view baseline techniques. In addition, we carefully design and generate a large synthetic training data set using physically-based rendering. During testing, our network takes only the raw glossy images as input, without extra information such as segmentation masks or lighting estimation. Results demonstrate that multi-view reconstruction can be significantly improved using the images filtered by our network. We also show promising performance on real world training and testing data.

Specular Detection on Glossy Surface Using Geometric Characteristics of Specularity in Top-View Images

“…They could be applied to classify specular regions based on trained features of convolutional neural networks. However, aside from the computational issue, the deep learning-based methods have a critical drawback that they can produce unexpected results for the input with different characteristics from the training dataset [ 37 , 38 ]. The threshold-based approaches are suitable for the pre-processing, due to their low computational cost.…”

Weakly Supported Plane Surface Reconstruction via Plane Segmentation Guided Point Cloud Enhancement

“…Given only images, Büyükatalay et al [57] directly filter the highlight surface, and Mallick et al [28] separate the specular reflection effects for surfaces that can be modeled with dichromatic reflectance. Wu et al [29] extend a ''specular to diffuse'' generative adversarial network translation for transforming objects with specular reflection into diffuse objects. Since these methods target only one type of weakly supported surfaces, they cannot fully solve the problem raised in this paper.…”

“…Most of the methods reported in the literature concentrate on flossy surface reconstruction and address the problem by adding extra hardware (e.g., coded pattern projection [25] and a two-layer LCD [26]), filtering the non-Lambert region [27], [28] or translating multi-view images of the objects with specular reflection to diffuse images [29]. Although these methods have made great progress towards non-diffuse surface reconstruction, they cannot handle textureless images, making them unable to realize general weakly supported surface reconstruction.…”

Temporally Coherent General Dynamic Scene Reconstruction

“…Research on multiple view dense dynamic reconstruction has primarily focused on indoor scenes with controlled illumination and static backgrounds, extending methods for multiple view reconstruction of static scenes (Seitz et al 2006) to sequences (Tung et al 2009). Deep learning based approaches have been introduced to estimate shape of dynamic objects from minimal camera views in constrained environment (Huang et al 2018;Wu et al 2018) and for rigid objects (Stutz and Geiger 2018). In the last decade, focus has shifted to more challenging outdoor scenes captured with both static and moving cameras.…”