Reflectance and Fluorescent Spectra Recovery Based on Fluorescent Chromaticity Invariance under Varying Illumination

Abstract

In recent years, fluorescence analysis of scenes has received attention. Fluorescence can provide additional information about scenes, and has been used in applications such as camera spectral sensitivity estimation, 3D reconstruction, and color relighting. In particular, hyperspectral images of reflective-fluorescent scenes provide a rich amount of data. However, due to the complex nature of fluorescence, hyperspectral imaging methods rely on specialized equipment such as hyperspectral cameras and specialized illuminants. In this paper, we propose a more practical approach to hyperspectral imaging of reflectivefluorescent scenes using only a conventional RGB camera and varied colored illuminants. The key idea of our approach is to exploit a unique property of fluorescence: the chromaticity of fluorescence emissions are invariant under different illuminants. This allows us to robustly estimate spectral reflectance and fluorescence emission chromaticity. We then show that given the spectral reflectance and fluorescent chromaticity, the fluorescence absorption and emission spectra can also be estimated. We demonstrate in results that all scene spectra can be accurately estimated from RGB images. Finally, we show that our method can be used to accurately relight scenes under novel lighting.

Simultaneous Surface Reflectance and Fluorescence Spectra Estimation

“…First, we adapted the nuclear norm minimization approach of Suo et al [37], (Supplemental Material, Appendix D). Second we implemented the multi-step algorithm of Fu et al [9]. These implementations, along with our algorithm, are available in our code repository.…”

“…Our methods have smaller error (RMSE) compared to these algorithms. Fu et al [9], [11] (Table 3) use a sequence of optimizations while the single fluorophore and chromaticity invariant (CIM) methods solve with a single optimization step. Perhaps the performance improvement is because the single step avoids accumulating errors across different stages.…”

“…Lam and Sato first estimate the reflectance component at about ten different wavelengths and then derive fluorescence components by subtracting the estimated reflectance contribution from the measurements. Fu, Lam et al [9], [11] acquire a sequence of RGB images under nine different, narrowband illuminants. They develop a multi-step algorithm that estimates the chromaticity coordinates of the reflectance and fluorescence spectra and then searches a fluorophore database arXiv:1605.04243v1 [cs.CV] 13 May 2016 Within-band ratios Reflectance, one fluorophore, spectra Fu et al [10] ∼ 30 bands Sinusoidal, 2 Within-band ratios Reflectance, one fluorophore, spectra Zheng et al [47] ∼ 30 bands Broadband, 3 Sequence of optimizations Reflectance, one fluorophore, spectra Lam et al [23] ∼ 30 bands Narrowband, ∼ 10 Multistep estimation Reflectance, one fluorophore, spectra Suo et al [37] ∼ 10 bands Narrowband, ∼ 6 Biconvex optimization Reflectance, multi fluorophore, intensity Zhang et al [46] 3 bands Broadband, 2 Ind.…”

“…They develop a multi-step algorithm that estimates the chromaticity coordinates of the reflectance and fluorescence spectra and then searches a fluorophore database arXiv:1605.04243v1 [cs.CV] 13 May 2016 Within-band ratios Reflectance, one fluorophore, spectra Fu et al [10] ∼ 30 bands Sinusoidal, 2 Within-band ratios Reflectance, one fluorophore, spectra Zheng et al [47] ∼ 30 bands Broadband, 3 Sequence of optimizations Reflectance, one fluorophore, spectra Lam et al [23] ∼ 30 bands Narrowband, ∼ 10 Multistep estimation Reflectance, one fluorophore, spectra Suo et al [37] ∼ 10 bands Narrowband, ∼ 6 Biconvex optimization Reflectance, multi fluorophore, intensity Zhang et al [46] 3 bands Broadband, 2 Ind. Component Analysis Chromaticity, intensity Fu et al [9] 3 bands Narrowband, ∼ 11 to determine the likely emission spectrum. Finally, Suo, Bian et al [37] estimate the reflectance spectrum and the Donaldson matrix, describing fluorescence properties of a surface, through nuclear norm minimization using Alternating Direction Minimization (ADM).…”

“…We present a collection of computational separation methods to simultaneously estimate the reflectance, excitation and emission spectra of a surface. This framework builds on several prior approaches [9], [11], [37], [47]. The methods also extend the active illumination reflection estimation methods [31], [32], and the general case of fluorescence unmixing methods when surface spectral properties are unknown [2], [19].…”