SPLINE-Net: Sparse Photometric Stereo Through Lighting Interpolation and Normal Estimation Networks

Abstract

This paper solves the Sparse Photometric stereo through Lighting Interpolation and Normal Estimation using a generative Network (SPLINE-Net). SPLINE-Net contains a lighting interpolation network to generate dense lighting observations given a sparse set of lights as inputs followed by a normal estimation network to estimate surface normals. Both networks are jointly constrained by the proposed symmetric and asymmetric loss functions to enforce isotropic constrain and perform outlier rejection of global illumination effects. SPLINE-Net is verified to outperform existing methods for photometric stereo of general BRDFs by using only ten images of different lights instead of using nearly one hundred images.

Recovering Surface Normal and Arbitrary Images: A Dual Regression Network for Photometric Stereo

“…For better estimating the non-Lambertian objects and taking full advantage of the information embedded in the neighborhood, subsequent methods were improved by applying the convolutional neural networks (CNN) [11]- [13], [37], [38]. The works [13] and [39] proposed a fully-convolutional network (FCN) to regress the surface normal, and a channel max-pooling operation was adopted to ensure the arbitrary number of input images.…”

Lightweight Photometric Stereo for Facial Details Recovery

“…Ikehata [22] presented an observation map to describe pixel-wise illumination information, and estimated surface normals with the observation map as input to an end-to-end convolutional network. Furthermore, Zheng et al [47] and Li et al [26] solved the sparse photometric stereo problem based on the observation map. This type of work assumes lighting directions as prior and cannot handle unknown lighting directions.…”

Deep Photometric Stereo Network with Multi-Scale Feature Aggregation

“…This architecture consists of a photometric stereo network to regress surface normals and an image reconstruction network to reconstruct the observed images in an inverse rendering manner [ 45 ]. For a sparse photometric stereo, Li et al introduced a trainable two-dimensional occlusion layer to manage cast shadows in the CNN framework [ 44 ], and Zheng et al presented a lighting interpolation network to yield a dense observation map from a sparse set of input and illumination pairs [ 46 ]. Chen et al presented a two-stage deep learning architecture, called a self-calibrating photometric stereo network (SDPS-Net), consisting of a lighting calibration network (LCNet) and a normal estimation network (NENet).…”

“…Capturing the ground-truth surface normals of real-world scenes with complex shapes and spatially varying materials at a large scale is extremely challenging. To mitigate this challenge, recent works have adopted a method of training networks on large-scale and physically meaningful synthetic datasets as an alternate way [ 40 , 41 , 42 , 43 , 44 , 45 , 46 ]. The proposed NENets are trained on publicly available synthetic photometric stereo datasets used in the training procedure of PS-FCN and SDPS-Net.…”

“…Deep learning algorithms have recently achieved remarkable progress in various domains, such as computer vision, speech recognition, and natural language processing. Following this trend, recent advances in the calibrated and uncalibrated photometric stereo to produce a high-fidelity surface normal have also been achieved employing deep learning, where the deep photometric stereo network learns the mapping from the multiple images to the surface normal vector [ 40 , 41 , 42 , 43 , 44 , 45 , 46 ].…”