Interpolant-based demosaicing routines for dual-mode visible/near-infrared imaging systems

Abstract

Dual-mode visible/near-infrared imaging systems, including a bioinspired six-channel design and more conventional four-channel implementations, have transitioned from a niche in surveillance to general use in machine vision. However, the demosaicing routines that transform the raw images from these sensors into processed images that can be consumed by humans or computers rely on assumptions that may not be appropriate when the two portions of the spectrum contribute different information about a scene. A solution can be found in a family of demosaicing routines that utilize interpolating polynomials and splines of different dimensionalities and orders to process images with minimal assumptions.

Decoupling channel count from field of view and spatial resolution in single-sensor imaging systems for fluorescence image-guided surgery

“… 28 , consistent with the methods in Ref. 38 . (c) A bioinspired long-pass/short-pass camera (or bioinspired camera) makes two substitutions compared with an RGB-IR camera: First, it replaces the single layer of silicon photodiodes with three layers of such photodiodes (each sensitive to shorter wavelength, longer wavelengths, or intermediate wavelengths), and second, it replaces the layer of four different absorptive filters with a layer of two different interferences filters (each serving as a short-pass filter tuned for VIS light or a long-pass filter tuned for NIR light).…”

“…A family of demosaicing routines that is compatible with dual-mode visible/near-infrared imaging systems like the RGB-IR sensor and the bioinspired sensor but does not make assumptions on the correlation between the visible channels and the near-infrared channels has already been proposed in Ref. 38 . These demosaicing routines recognize that each channel is fully defined by a complicated function of two variables, the row number and column number, that can be locally approximated by much simpler interpolating polynomials taken in one of several variations.…”

“…This comparison can be done with the sum of absolute errors, taken over all pixels, which accounts for both the vanishing error from known pixels captured by the imaging system and the nonvanishing error from unknown pixels filled in by the demosaicing routine, or it can be done with the mean of absolute errors, taken over unknown pixels, which downplays the role performed by the imaging system in favor of that performed by the demosaicing routine. 38 Since the variation in these two measures tracks the variation between the unmosaiced image and the demosaiced image in a simple and proportional manner, they provide some transparency into how a computer program developing a surgical strategy will be affected by the combined effect of the imaging system and demosaicing routine. Alternatively, a comparison of two imaging systems or demosaicing routines can be facilitated by a comparison of the perceived deviation between the unmosaiced image and the demosaiced images.…”

“…This can be done with the 99th percentile of the structural dissimilarity index, taken over unknown pixels, which describes the change in texture in the vicinity of those pixels filled in by the demosaicing routine, or the 99th percentile of the color difference, also taken over unknown pixels, which describes the change in color at those pixels. 38 Since the variation in these two measures tracks the variation between the unmosaiced image and the demosaiced image in a complex but perceptually consistent manner, they provide a window into how a human observer thinking through their own surgical approach will be affected as an image is captured and processed.…”

“… 28 That being said, the 99th percentile of the structural dissimilarity can be pooled across the visible channels or the near-infrared channels so that it can be defined regardless of the channel count, whereas the 99th percentile of the color difference must be computed over three visible channels or three near-infrared channels so that it is only defined for a trichromatic channel configuration. 38 The structural dissimilarity can thus be presented for both the visible channels and the near-infrared channels from both the RGB-IR sensor and the bioinspired sensor, but the color difference must only be presented for the visible channels from the RGB-IR sensor in addition to the visible channels and near-infrared channels from the bioinspired sensor. In this way, the structural dissimilarity for the near-infrared channels describes the change in texture perceived in those channels by a human operator that is viewing them in either a monochromatic format or a trichromatic format, whereas the color difference for the near-infrared channels describes the change in color perceived in those channels by a human operator viewing them in a trichromatic format.…”