Pixel-Aware Deep Function-Mixture Network for Spectral Super-Resolution

Abstract

Spectral super-resolution (SSR) aims at generating a hyperspectral image (HSI) from a given RGB image. Recently, a promising direction is to learn a complicated mapping function from the RGB image to the HSI counterpart using a deep convolutional neural network. This essentially involves mapping the RGB context within a size-specific receptive field centered at each pixel to its spectrum in the HSI. The focus thereon is to appropriately determine the receptive field size and establish the mapping function from RGB context to the corresponding spectrum. Due to their differences in category or spatial position, pixels in HSIs often require different-sized receptive fields and distinct mapping functions. However, few efforts have been invested to explicitly exploit this prior.To address this problem, we propose a pixel-aware deep function-mixture network for SSR, which is composed of a new class of modules, termed function-mixture (FM) blocks. Each FM block is equipped with some basis functions, i.e., parallel subnets of different-sized receptive fields. Besides, it incorporates an extra subnet as a mixing function to generate pixel-wise weights, and then linearly mixes the outputs of all basis functions with those generated weights. This enables us to pixel-wisely determine the receptive field size and the mapping function. Moreover, we stack several such FM blocks to further increase the flexibility of the network in learning the pixel-wise mapping. To encourage feature reuse, intermediate features generated by the FM blocks are fused in late stage, which proves to be effective for boosting the SSR performance. Experimental results on three benchmark HSI datasets demonstrate the superiority of the proposed method.

Brightness Invariant Deep Spectral Super-Resolution

“…In order to evaluate the proposed methodology, distinct network architectures from the current state-of-the-art methods are considered: the HSCNN+R [ 17 ], an adopted UNet [ 18 ], the adaptive weight attention network (AWAN) [ 15 ] and the pixel-aware deep function-mixture network (FMNet) [ 13 ]. The respective code is publicly available for all individual network architectures.…”

“…The AWCA module has a reduction ratio of and the PSNL model an r -value of 8. FMNet The utilized configuration [ 13 ] consists of two FM blocks with each block containing three basis functions. Each basis function as well as the mixing functions are formed by two convolutional blocks having 64 feature maps.…”

“…The recent NTIRE2020 challenge on spectral reconstruction from RGB [ 11 ] as well as its predecessor in 2018 [ 12 ] provide a concise overview over the state-of-the-art methods. In summary, the best performing methods in terms of spectral reconstruction quality exclusively consist of complex convolutional neural networks [ 13 , 14 , 15 , 16 , 17 , 18 , 19 ]. A major limitation of deep learning based approaches for SSR is their susceptibility to a varying brightness, i.e., signal scale.…”

Hyperspectral Remote Sensing Imagery Generation From RGB Images Based on Joint Discrimination

“…The training process of R2HGAN takes approximately 19 hours. Under the same conditions, we compare R2HGAN to four spectral super resolution methods, including MsCNN [51], HSCNN+ [56], FMNet [84] and HSRNet [85]. MsCNN and HSCNN+ are based on U-Net and DenseNet respectively.…”

Low to High Dimensional Modality Hallucination Using Aggregated Fields of View

“…They use pix2pix [10] as well for generating RGB from semantic labels and we empirically show that it doesn't work well in our case. Work done in [11] and [12] aims to solve different problem statements but is related to our work with regards to methodology and motivation. They are functionally different from ours as the features obtained from different kernels are fused only at the end.…”