Mosaic Convolution-Attention Network for Demosaicing Multispectral Filter Array Images

Feng, Kai; Zhao, Yongqiang; Chan, Jonathan Cheung-Wai; Kong, Seong G.; Zhang, Xun; Wang, Binglu

doi:10.1109/tci.2021.3102052

Cited by 41 publications

(16 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The advance in CFA design for RGB cameras sparks great interests in the CFA design for MS cameras [55,45,43,44,1,24,38,47,15,6,63,56]. An early work by Ramanath et al presented a CFA pattern which is composed of seven bands and they are arranged hexagonally [55].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Jointly Learning Band Selection and Filter Array Design for Hyperspectral Imaging

Dai²,

Gool

2023

2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

A single-shot multispectral camera equipped with an optimized color filter array (CFA) has the potential to deliver a fast and low-cost hyperspectral (HS) imaging system. Previous solutions are largely restricted to designing demosaicing algorithms for fixed CFAs -be it the Bayer color pattern or evenly-spaced spectral multiplexing patterns. Since sampling and reconstruction are tightly-coupled, the ability to search for an optimal solution is severely constrained by using predefined CFAs. In this work, we simultaneously address the problem of spectral band selection, CFA design, image demosaicing, and spectral image recovery in a joint learning framework for single-shot HS imaging. We propose a reinforcement learning (RL) based method for spectral band selection and a novel neural network for CFA generation, image demosaicing, and HS image recovery. The final spectral reconstruction accuracy is used to supervise the training of the main network to maximize the synergies between those tightly-related tasks. The RL method regards the main network as an agent to collect reward. Our final method delivers a simple setup -as simple as an RGB camera -for HS imaging. Experimental results show that our method outperforms competing methods by a large margin.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Demosaicing. Demosaicing has been a well-established field with many great works proposed [23,29,35,47,15,6,5]. The general idea is to use the measured signals at sparse locations to fill up the missing values in neighboring locations.…”

Section: Related Workmentioning

confidence: 99%

Jointly Learning Band Selection and Filter Array Design for Hyperspectral Imaging

Dai²,

Gool

2023

2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

show abstract

“…Of course reconstruction algorithms for different types of hyperspectral cameras can be easily evaluated like demosaicing for multispectral filter arrays [46]. Combinations of these techniques also can be evaluated, e.g., a mixture of multispectral filter arrays and camera arrays.…”

Section: Other Applicationsmentioning

confidence: 99%

Synthetic hyperspectral array video database with applications to cross-spectral reconstruction and hyperspectral video coding

Sippel¹,

Seiler²,

Kaup³

2023

J. Opt. Soc. Am. A

View full text Add to dashboard Cite

In this paper, a synthetic hyperspectral video database is introduced. Since it is impossible to record ground truth hyperspectral videos, this database offers the possibility to leverage the evaluation of algorithms in diverse applications. For all scenes, depth maps are provided as well to yield the position of a pixel in all spatial dimensions as well as the reflectance in spectral dimension. Two novel algorithms for two different applications are proposed to prove the diversity of applications that can be addressed by this novel database. First, a cross-spectral image reconstruction algorithm is extended to exploit the temporal correlation between two consecutive frames. The evaluation using this hyperspectral database shows an increase in PSNR of up to 5.6 dB dependent on the scene. Second, a hyperspectral video coder is introduced which extends an existing hyperspectral image coder by exploiting temporal correlation. The evaluation shows rate savings of up to 10% depending on the scene. The novel hyperspectral video database and source code is available at https:// github.com/ FAU-LMS/ HyViD for use by the research community.

show abstract

“…Research on multispectral image demosaicing has been conducted in various studies [1,5,20,24,25,38,39,41,42]. Current methods for demosaicing can be categorized into interpolation-based, matrix-factorization/recovery-based, and deep learning approaches.…”

Section: Hsi Demosaicingmentioning

confidence: 99%

MSFA-Frequency-Aware Transformer for Hyperspectral Images Demosaicing

Zeng¹,

Feng²,

Shaoguang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Hyperspectral imaging systems that use multispectral filter arrays (MSFA) capture only one spectral component in each pixel. Hyperspectral demosaicing is used to recover the non-measured components. While deep learning methods have shown promise in this area, they still suffer from several challenges, including limited modeling of non-local dependencies, lack of consideration of the periodic MSFA pattern that could be linked to periodic artifacts, and difficulty in recovering high-frequency details. To address these challenges, this paper proposes a novel demosaicing framework, the MSFA-frequency-aware Transformer network (FDM-Net). FDM-Net integrates a novel MSFA-frequency-aware multi-head self-attention mechanism (MaFormer) and a filter-based Fourier zero-padding method to reconstruct high pass components with greater difficulty and low pass components with relative ease, separately. The advantage of Maformer is that it can leverage the MSFA information and non-local dependencies present in the data. Additionally, we introduce a joint spatial and frequency loss to transfer MSFA information and enhance training on frequency components that are hard to recover. Our experimental results demonstrate that FDM-Net outperforms state-of-the-art methods with 6dB PSNR, and reconstructs high-fidelity details successfully.

show abstract

Mosaic Convolution-Attention Network for Demosaicing Multispectral Filter Array Images

Cited by 41 publications

References 60 publications

Jointly Learning Band Selection and Filter Array Design for Hyperspectral Imaging

Jointly Learning Band Selection and Filter Array Design for Hyperspectral Imaging

Synthetic hyperspectral array video database with applications to cross-spectral reconstruction and hyperspectral video coding

MSFA-Frequency-Aware Transformer for Hyperspectral Images Demosaicing

Contact Info

Product

Resources

About