Depth Super-Resolution Meets Uncalibrated Photometric Stereo

Abstract

Input HR RGB images and LR depth mapsOutput HR albedo and depth maps Relighting Figure 1: Given an RGB-D sequence of n ≥ 4 low-resolution (320 × 240 px) depth maps and high-resolution (1280 × 1024 px) RGB images acquired from the same viewing angle but under varying, unknown lighting, high-resolution depth and reflectance maps are estimated by combining super-resolution and photometric stereo within a variational framework. AbstractA novel depth super-resolution approach for RGB-D sensors is presented. It disambiguates depth super-resolution through high-resolution photometric clues and, symmetrically, it disambiguates uncalibrated photometric stereo through low-resolution depth cues. To this end, an RGB-D sequence is acquired from the same viewing angle, while illuminating the scene from various uncalibrated directions. This sequence is handled by a variational framework which fits high-resolution shape and reflectance, as well as lighting, to both the low-resolution depth measurements and the high-resolution RGB ones. The key novelty consists in a new PDE-based photometric stereo regularizer which implicitly ensures surface regularity. This allows to carry out depth super-resolution in a purely data-driven manner, without the need for any ad-hoc prior or material calibration. Realworld experiments are carried out using an out-of-the-box RGB-D sensor and a hand-held LED light source.

Variational Uncalibrated Photometric Stereo Under General Lighting

“…However, the previous approaches to photometric stereo under natural illumination assume calibrated lighting, where calibration is deduced from time and GPS coordinates or from a calibration target. The case of both general and uncalibrated lighting is much more challenging and has been fewly explored, apart from studies restricted to sparse 3D-reconstructions [46] or relying on the prior knowledge of a rough geometry [4,27,40,47]. Uncalibrated photometric stereo under natural illumination has been revisited recently in [34], using a spatially-varying equivalent directional lighting model.…”

“…Having fixed the choice of (V, µ, M ), we can now evaluate our approach against other state-of-the-art methods. We compare our results against those obtained by an uncalibrated photometric stereo approach assuming directional lighting [15], and another one assuming general (first-order spherical harmonics) illumination yet relying on an input shape prior (e.g., from an RGB-D sensor) [40]. As this limiting assumption on the access to a sensor-based depth prior is not always given and to make comparison fair, we input as depth prior to this method the ballooning initialization described in Section 5.1.…”

“…Despite relying on a directional lighting model, the approach of [15] produces reasonable results on some datasets (Face1, Ovenmitt or Shirt), but it fails on others. As [40] assumes a reliable prior on depth in order to perform a photometric refinement, this approach is biased towards its initialization and thus, only when the depth prior is very close to Figure 7. Results of state-of-the-art approaches and our approach on two out of the 36 synthetic datasets.…”

“…Thus, color changes in the images are caused by lighting only, as depicted in model (1) and (7) in the main paper. Table 2 shows the mean angular error (MAE) of each dataset on the state-of-the-art approaches [15,34,40] and our proposed methodology. It can be seen that our approach consistently overcomes [15,34,40] by a factor of 2-3.…”

“…While [15] gives more meaningful results on Armadillo with Constant albedo, depth deteriorates strongly on Lucy with Hippie albedo. Methods of [34,40] both result in rather flattened shapes (cf. Lucy).…”