Visible and NIR microscopic hyperspectrum reconstruction from RGB images with deep convolutional neural networks

Abstract

We investigate the microscopic hyperspectral reconstruction from RGB images with a deep convolutional neural network (DCNN) in this paper. Based on the microscopic hyperspectral imaging system, a homemade dataset consisted of microscopic hyperspectral and RGB image pairs is constructed. For considering the importance of spectral correlation between neighbor spectral bands in microscopic hyperspectrum reconstruction, the 2D convolution is replaced by 3D convolution in the DCNN framework, and a metric (weight factor) used to evaluate the performance reconstructed hyperspectrum is also introduced into the loss function used in training. The effects of the dimension of convolution kernel and the weight factor in the loss function on the performance of the reconstruction model are studied. The overall results indicate that our model can show better performance than the traditional models applied to reconstruct the hyperspectral images based on DCNN for the public and the homemade microscopic datasets. In addition, we furthermore explore the microscopic hyperspectrum reconstruction from RGB images in infrared region, and the results show that the model proposed in this paper has great potential to expand the reconstructed hyperspectrum wavelength range from the visible to near infrared bands.