Snapshot multispectral imaging using a diffractive optical network

Abstract

Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal-plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72λm, where λm is the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially repeating virtual spectral filter array with 2 × 2 = 4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.

Optical color routing enabled by deep learning

“…More impressively, as demonstrated in Fig. 2f, the pixel-level Bayer-type color router holds the color collection efficiencies of 58,59, and 49% at wavelengths of 640, 540, and 460 nm, representing R, G and B pixels, respectively. Using a 200 μm sized array working with a monochromatic imaging sensor, the image intensity reaches twice that achieved using a Bayer color filter.…”

“…In Fig. 4f, a proposed convolutional correla- 58 Copyright © 2023 Springer Nature. (b) A method using artificial neural networks to approximate light scattering by multilayer nanoparticles.…”

Polarization-based all-optical logic gates using diffractive neural networks

“…During the training process, the design of the passive diffractive layers, or neurons, is optimized such that the network performs a specific function. D 2 NN has been applied to image recognition [29][30][31][32][33][35][36][37][38][39][40][41][42][43][44][45][46], optical logic operations [21,47,48], terahertz pulse shaping [49], phase retrieval [50], and image reconstruction [34,[51][52][53] etc.…”

Advanced all-optical classification using orbital-angular-momentum-encoded diffractive networks

“…Each pixel point on the diffractive layer is a parameter that can be learned by the computer and can be used for independent complex-valued tuning of the light field. Based on its capabilities in optical information processing, normalD2NN has been applied to image recognition, 11 , 26 – 40 optical logic operations, 41 – 43 terahertz pulse shaping, 44 phase retrieval, 45 and image reconstruction, 15 , 46 – 48 etc.…”

“…Diffractive networks performed in passive optical elements have the advantages of fast processing speed and low energy consumption, while also enabling flexible utilization of various degrees of freedom of light in the network. For example, when using broadband light instead of monochromatic light to illuminate the diffractive networks, spectrally encoded machine vision applications, 15 , 38 parallel computing, 39 snapshot multispectral imaging, 48 and spatially controlled wavelength multiplexing/demultiplexing 49 can be accomplished. In addition, the linear transformation of polarization multiplexing can be achieved by using the polarization properties of light in diffractive networks instead of being based on birefringence or polarization-sensitive materials, 50 which fully demonstrates the classification and computational potential of diffractive networks in complex-valued matrix vector operations.…”