Binary Tree-based Generic Demosaicking Algorithm for Multispectral Filter Arrays

Abstract

In this paper, we extend the idea of using mosaicked color filter array (CFA) in color imaging, which has been widely adopted in the digital color camera industry, to the use of multispectral filter array (MSFA) in multispectral imaging. The filter array technique can help reduce the cost, achieve exact registration, and improve the robustness of the imaging system. However, the extension from CFA to MSFA is not straightforward. First, most CFAs only deal with a few bands (3 or 4) within the narrow visual spectral region, while the design of MSFA needs to handle the arrangement of multiple bands (more than 3) across a much wider spectral range. Second, most existing CFA demosaicking algorithms assume the fixed Bayer CFA and are confined to properties only existed in the color domain. Therefore, they cannot be directly applied to multispectral demosaicking. The main challenges faced in multispectral demosaicking is how to design a generic algorithm that can handle the more diversified MSFA patterns, and how to improve performance with a coarser spatial resolution and a less degree of spectral correlation. In this paper, we present a binary tree based generic demosaicking method. Two metrics are used to evaluate the generic algorithm, including the root mean-square error (RMSE) for reconstruction performance and the classification accuracy for target discrimination performance. Experimental results show that the demosaicked images present low RMSE (less than 7) and comparable classification performance as original images. These results support that MSFA technique can be applied to multispectral imaging with unique advantages.

A Linear Interpolation Algorithm for Spectral Filter Array Demosaicking

“…For the sake of comparison, we also implemented, in addition to the proposed method, another 5 techniques of multispectral demosaicking including bilinear interpolation [18], channel difference interpolation extended from the color difference method [13], LMMSE method [7] based on constant difference, binary tree based demosaicking [10] and discrete wavelet transformed based approach [15].…”

“…The assumptions on which a variety of CFA demosaicking methods are based are very often not applicable to the multispectral domain [10]. As an example, the luminance-chrominance color space used for CFA pattern is senseless for SFA demosaicking, as a result of the difficulties in extracting luminance and chrominance information separately in multispectral images.…”

N-LMMSE Demosaicing for Spectral Filter Arrays

“…We simulate acquisition by a real sensor( Figure 6) that spans visible and NIR, 10,70 where spatial layout follows Miao binary tree. 36,71,72 The benchmark demosaicing, Miao binary tree, 34 is applied on the raw data for reconstruction. Algorithms using guided filter and adaptive kernel 11 are tuned to SFAs having quincunx sampling of green so it would be non-efficient and unfair to compare with our instance of SFA.…”

“…Extension of the difference between channels have been considered. 32,33 Bilinear interpolation have also been implemented and upgraded into the binary-tree approach from Miao et al, [34][35][36] as well as the vector median interpolation. 37 Linear general formulations have shown great results, such as the spectral linear minimum mean square error (LMMSE), 38,39 RGB-NIR LMMSE, 40 linear reconstruction by training, [41][42][43] linear intensity-based demosaicing 44 and inter-channel correlation-based demosaicing 45 have been studied.…”

Energy balance in Spectral Filter Array camera design

“…Various research directions have been explored: filter characteristics [8][9][10][11], adequate full resolution image processing, e.g. demosaicing [12][13][14][15][16][17], and other aspects [18,19].…”