Improved U-Net3+ With Spatial–Spectral Transformer for Multispectral Image Reconstruction

Abstract

Multispectral image reconstruction, which aims to recover a three-dimensional (3D) spatial-spectral signal from a two-dimensional measurement in a spectral camera based on ghost imaging via sparsity constraint (GISC), has been attracting much attention recently. However, faced with abundant 3D spectral data, the reconstruction quality cannot meet the visual requirements. Based on the robust data processing capability of deep learning, a novel network called SSTU-Net3+ is constructed by improving U-Net3+ with a spatial-spectral transformer (SST). To enhance the feature representation of images during reconstruction, mixed pooling modules and new convolution processes are proposed to improve the performance of the encoder and decoder, with U-Net3+ as the backbone. To boost the quality of reconstructed images, with split and concatenate (Concat) operations, we construct SST modules by exploiting both spatial and spectral correlations of multispectral images to refine the spatial and spectral features. Furthermore, we employ the SST in the decoder to reconstruct the desired 3D cube. Given similar network parameters, experiments on GISC spectral imaging data show that, compared to convolutional neural network-based methods, the average peak signal-to-noise ratio of images reconstructed using SSTU-Net3+ is improved by 3%, the structural similarity is enhanced by 3%, and the spectral angle mapping is cut by 12%. Particularly, compared to differential ghost imaging and compressed sensing, the reconstruction quality of SSTU-Net3+ has been significantly improved. SSTU-Net3+ can process a large amount of 3D multispectral image data more efficiently and construct the target image more accurately than the abovementioned methods. 

Taming Transformers for High-Resolution Image Synthesis

“…Our d = 38 model outperforms current proprietary(Betker et al, 2023; ide, 2024) and open(Sauer et al, 2023; pla, 2024;Chen et al, 2023;Pernias et al, 2023) SOTA generative image models in human preference evaluation on the…”

“…Our largest models outperform state-of-the art open models such as SDXL (Podell et al, 2023), SDXL-Turbo (Sauer et al, 2023), Pixart-α (Chen et al, 2023), and closed-source models such as DALL-E 3 (Betker et al, 2023) both in quantitative evaluation (Ghosh et al, 2023) of prompt understanding and human preference ratings.…”

“…For diffusion models, U-Net architectures (Ronneberger et al, 2015) have been the dominant choice (Ho et al, 2020;Rombach et al, 2022;Balaji et al, 2022). While some recent works explore diffusion transformer backbones (Peebles & Xie, 2023;Chen et al, 2023;Ma et al, 2024), scaling laws for text-to-image diffusion models remain unexplored.…”

“…MM-DiT is specifically designed to handle different domains, here text and image tokens, using (two) different sets of trainable model weights. More specifically, we follow the experimental setup from Section 5.1 and compare text-to-image performance on CC12M of DiT, CrossDiT (DiT but with cross-attending to the text tokens instead of sequence-wise concatenation (Chen et al, 2023)) and our MM-DiT. For MM-DiT, we compare models with two sets of weights and three sets of weights, where the latter handles the CLIP (Radford et al, 2021) and T5 (Raffel et al, 2019) tokens (c.f .…”

High-Quality Multispectral Image Reconstruction for the Spectral Camera Based on Ghost Imaging via Sparsity Constraints Using CoT-Unet

“…Meanwhile, the lower the system's sampling rate, the more complex reconstructing a clear target image is. SSTU-Net3+ [13] obtained relatively good reconstruction results, but no further discussion was made on the sampling rate. Although existing deep learning methods have accepted good results in multispectral image reconstruction tasks, there are still the following problems in introducing them to GISC spectral imaging:…”