Hyperspectral (HS) imaging provides rich spatial and spectral information and extends image inspection beyond human perception. Existing approaches, however, suffer from several drawbacks such as low sensitivity, resolution and/or frame rate, which confines HS cameras to scientific laboratories. Here we develop a video-rate HS camera capable of collecting spectral information on real-world scenes with sensitivities and spatial resolutions comparable with those of a typical RGB camera. Our camera uses compressive sensing, whereby spatial–spectral encoding is achieved with an array of 64 complementary metal–oxide–semiconductor (CMOS)-compatible Fabry–Pérot filters placed onto a monochromatic image sensor. The array affords high optical transmission while minimizing the reconstruction error in subsequent iterative image reconstruction. The experimentally measured sensitivity of 45% for visible light, the spatial resolution of 3 px for 3 dB contrast, and the frame rate of 32.3 fps at VGA resolution meet the requirements for practical use. For further acceleration, we show that AI-based image reconstruction affords operation at 34.4 fps and full high-definition resolution. By enabling practical sensitivity, resolution and frame rate together with compact size and data compression, our HS camera holds great promise for the adoption of HS technology in real-world scenarios, including consumer applications such as smartphones and drones.

This paper presents system-level analysis of a sensor capable of simultaneously acquiring both standard absorption based RGB color channels (400-700nm, ~75nm FWHM), as well as an additional NIR channel (central wavelength: ~808 nm, FWHM: ~30nm collimated light). Parallel acquisition of RGB and NIR info on the same CMOS image sensor is enabled by monolithic pixel-level integration of both a NIR pass thin film filter and NIR blocking filters for the RGB channels. This overcomes the need for a standard camera-level NIR blocking filter to remove the NIR leakage present in standard RGB absorption filters from ~700-1000nm. Such a camera-level NIR blocking filter would inhibit the acquisition of the NIR channel on the same sensor. Thin film filters do not operate in isolation. Rather, their performance is influenced by the system context in which they operate. The spectral distribution of light arriving at the photo diode is shaped a.o. by the illumination spectral profile, optical component transmission characteristics and sensor quantum efficiency. For example, knowledge of a low quantum efficiency (QE) of the CMOS image sensor above 800nm may reduce the filter's blocking requirements and simplify the filter structure. Similarly, knowledge of the incoming light angularity as set by the objective lens' F/# and exit pupil location may be taken into account during the thin film's optimization. This paper demonstrates how knowledge of the application context can facilitate filter design and relax design trade-offs and presents experimental results.

No abstract

Recent developments in multispectral cameras have demonstrated how compact and low-cost spectral sensors can be made by monolithically integrating filters on top of commercially available image sensors. In this paper, the fabrication of a RGB + NIR variation to such a single-chip imaging system is described, including the integration of a metallic shield to minimize crosstalk, and two interference filters: a NIR blocking filter, and a NIR bandpass filter. This is then combined with standard polymer based RGB colour filters. Fabrication of this chip is done in imec’s 200 mm cleanroom using standard CMOS technology, except for the addition of RGB colour filters and microlenses, which is outsourced.