Physically Plausible Spectral Reconstruction

Abstract

Spectral reconstruction algorithms recover spectra from RGB sensor responses. Recent methods—with the very best algorithms using deep learning—can already solve this problem with good spectral accuracy. However, the recovered spectra are physically incorrect in that they do not induce the RGBs from which they are recovered. Moreover, if the exposure of the RGB image changes then the recovery performance often degrades significantly—i.e., most contemporary methods only work for a fixed exposure. In this paper, we develop a physically accurate recovery method: the spectra we recover provably induce the same RGBs. Key to our approach is the idea that the set of spectra that integrate to the same RGB can be expressed as the sum of a unique fundamental metamer (spanned by the camera’s spectral sensitivities and linearly related to the RGB) and a linear combination of a vector space of metameric blacks (orthogonal to the spectral sensitivities). Physically plausible spectral recovery resorts to finding a spectrum that adheres to the fundamental metamer plus metameric black decomposition. To further ensure spectral recovery that is robust to changes in exposure, we incorporate exposure changes in the training stage of the developed method. In experiments we evaluate how well the methods recover spectra and predict the actual RGBs and RGBs under different viewing conditions (changing illuminations and/or cameras). The results show that our method generally improves the state-of-the-art spectral recovery (with more stabilized performance when exposure varies) and provides zero colorimetric error. Moreover, our method significantly improves the color fidelity under different viewing conditions, with up to a 60% reduction in some cases.

On the Optimization of Regression-Based Spectral Reconstruction

“…Spectral reconstruction (SR) seeks to recover spectral information from the RGB data of a single camera [ 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 ]. Assuming the recovery error of SR is low enough (regarding the results in the literature and in this paper), we can essentially measure spectra using an RGB camera.…”

“…This approach hypothesizes that object-level descriptions of the RGBs can—though requiring much more computational resources—aid the recovery of spectra. However, DNN-based models do not always perform better than simple regressions [ 55 ] and often suffer from instability issue when recovering spectra at different brightness scales [ 56 , 57 , 58 ]. Furthermore, spectra recovered by DNNs are shown to be less accurate in color [ 57 , 59 ] and do not necessarily provide more accurate cross-illumination or cross-device color reproductions [ 57 ].…”

“…Most top DNN models are also designed to minimize MRAE directly [ 60 , 61 , 62 , 63 , 64 ]. However, all regressions used in SR are optimized by convention for Root Mean Square Error (RMSE) while evaluated using MRAE (or other similar relative errors) when compared with the DNNs [ 55 , 56 , 57 , 60 , 61 ]. In other words, these regressions are optimized for one metric, and then how well they work is evaluated with another.…”

Optimal selection of representative samples for efficient digital camera‐based spectra recovery

“…Many methods have been proposed to calculate the spectra recovery matrix, such as pseudoinverse, PCA, Wiener estimation, compressive sensing, and so on 16‐20 . In addition, deep learning based methods, such Convolutional Neural Network (CNN), U‐net, and Generative Adversarial Network (GAN)‐based methods, are also introduced into spectra recovery in recent years with fast development of computer visual technology 21‐23 . In this study, taking the commonly used pseudoinverse method as example, the spectra recovery matrix is calculated as in Equation (): where Q is the spectra recovery matrix, R train and D train are the spectral reflectance and camera response matrix of training set, and superscript + is the pseudoinverse operator.…”

Deep-Learning-Based Multispectral Image Reconstruction from Single Natural Color RGB Image—Enhancing UAV-Based Phenotyping

“…Higher-dimensional hyper-/multispectral information recovery from lower spectraldimensional ncRGB space is ill-posed in practice [14,21]. Various methods have been proposed to reconstruct HSIs from a single ncRGB image [6,22].…”