Self-Calibrating Deep Photometric Stereo Networks

Chen, Guanying; Han, Kai; Shi, Boxin; Matsushita, Yasuyuki; Wong, Kwan-Yee K.

doi:10.1109/cvpr.2019.00894

Cited by 115 publications

(173 citation statements)

References 29 publications

Supporting

Mentioning

158

Contrasting

Order By: Relevance

“…We have presented preliminarily results of this work in [19], [20], and this paper extends them in several aspects. First, we extend PS-FCN to handle surfaces with spatiallyvarying BRDFs (SVBRDFs) by introducing a simple yet effective data normalization strategy.…”

Section: Introductionmentioning

confidence: 71%

Deep Photometric Stereo for Non-Lambertian Surfaces

Chen

Han²,

Shi³

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

This paper addresses the problem of photometric stereo, in both calibrated and uncalibrated scenarios, for non-Lambertian surfaces based on deep learning. We first introduce a fully convolutional deep network for calibrated photometric stereo, which we call PS-FCN. Unlike traditional approaches that adopt simplified reflectance models to make the problem tractable, our method directly learns the mapping from reflectance observations to surface normal, and is able to handle surfaces with general and unknown isotropic reflectance. At test time, PS-FCN takes an arbitrary number of images and their associated light directions as input and predicts a surface normal map of the scene in a fast feed-forward pass. To deal with the uncalibrated scenario where light directions are unknown, we introduce a new convolutional network, named LCNet, to estimate light directions from input images. The estimated light directions and the input images are then fed to PS-FCN to determine the surface normals. Our method does not require a pre-defined set of light directions and can handle multiple images in an order-agnostic manner. Thorough evaluation of our approach on both synthetic and real datasets shows that it outperforms state-of-the-art methods in both calibrated and uncalibrated scenarios.

show abstract

Section: Introductionmentioning

confidence: 71%

Deep Photometric Stereo for Non-Lambertian Surfaces

Chen

Han²,

Shi³

et al. 2022

IEEE Trans. Pattern Anal. Mach. Intell.

Self Cite

View full text Add to dashboard Cite

show abstract

“…with Ω ⊂ R 2 the mask of the object to reconstruct, I i : Ω → R the i-th input graylevel image, ρ the reflectance (albedo) map, n the normal map (which encodes the 3Dgeometry), and s i ∈ R 3 a vector representing the incident lighting in the i-th image (in intensity and direction). Most of recent works on photometric stereo have focused on relaxing the assumptions of Lambertian reflectance (i.e., handling surfaces which exhibit a specular behavior) [4,21,11,29] and calibrated directional lighting (i.e., handling unknown or non-uniform lighting) [5,10,13,22], see for instance [23] for some discussion and [3] for a state-ofthe-art joint solution to both issues using deep neural networks. However, in all of these recent works the object to reconstruct is assumed to be segmented a priori: the whole pipeline relies on the knowledge of the domain Ω.…”

Section: Variational Methods For Photometric Stereo and Segmentationmentioning

confidence: 99%

Photometric Segmentation: Simultaneous Photometric Stereo and Masking

Haefner

Quéau

Cremers

2019

2019 International Conference on 3D Vision (3DV)

View full text Add to dashboard Cite

This work is concerned with both the 3D-reconstruction of an object using photometric stereo, and its 2Dsegmentation from the background. In contrast with previous works on photometric stereo which assume that a mask of the area of interest has been computed beforehand, we formulate 3D-reconstruction and 2D-segmentation as a joint problem. The proposed variational solution combines a differential formulation of photometric stereo with the classic Chan-Vese model for active contours. Given a set of photometric stereo images, this solution simultaneously infers a binary mask of the object of interest and a depth map representing its 3D-shape. Experiments on real-world datasets confirm the soundness of simultaneously solving both these classic computer vision problems, as the joint approach considerably simplifies the overall 3D-scanning process for the end-user.

show abstract

“…As in SfS, the problem must be reformulated globally, and the integrability constraint must be imposed [133]. But even then, a low-frequency ambiguity known as the generalised bas-relief ambiguity remains [12]: it is necessary to introduce additional priors, see [106] for an overview of existing uncalibrated photometric stereo approaches, and [19] for a modern solution based on deep learning. Another situation where PS is ill-posed is when only two images are considered [93]: in each pixel there exist two possible normals explaining the pair of graylevels, even with known reflectance and lighting.…”

Section: Well-posedness Of Psmentioning

confidence: 99%