Deep learning has been transforming our ability to execute advanced inference tasks using computers. Here we introduce a physical mechanism to perform machine learning by demonstrating an all-optical diffractive deep neural network (DNN) architecture that can implement various functions following the deep learning-based design of passive diffractive layers that work collectively. We created 3D-printed DNNs that implement classification of images of handwritten digits and fashion products, as well as the function of an imaging lens at a terahertz spectrum. Our all-optical deep learning framework can perform, at the speed of light, various complex functions that computer-based neural networks can execute; will find applications in all-optical image analysis, feature detection, and object classification; and will also enable new camera designs and optical components that perform distinctive tasks using DNNs.

Application-specific optical processors have been considered disruptive technologies for modern computing that can fundamentally accelerate the development of artificial intelligence (AI) by offering substantially improved computing performance. Recent advancements in optical neural network architectures for neural information processing have been applied to perform various machine learning tasks. However, the existing architectures have limited complexity and performance; and each of them requires its own dedicated design that cannot be reconfigured to switch between different neural network models for different applications after deployment. Here, we propose an optoelectronic reconfigurable computing paradigm by constructing a diffractive processing unit (DPU) that can efficiently support different neural networks and achieve a high model complexity with millions of neurons. It allocates almost all of its computational operations optically and achieves extremely high speed of data modulation and large-scale network parameter updating by dynamically programming optical modulators and photodetectors. We demonstrated the reconfiguration of the DPU to implement various diffractive feedforward and recurrent neural networks and developed a novel adaptive training approach to circumvent the system imperfections. We applied the trained networks for high-speed classifying of handwritten digit images and human action videos over benchmark datasets, and the experimental results revealed a comparable classification accuracy to the electronic computing approaches. Furthermore, our prototype system built with off-the-shelf optoelectronic components surpasses the performance of state-of-the-art graphics processing units (GPUs) by several times on computing speed and more than an order of magnitude on system energy efficiency. We believe the proposed reconfigurable DPU is a remarkable step towards high-performance neuromorphic optoelectronic computing processors that can achieve real-time dynamic architecture configurations according to software and will facilitate a broad range of AI applications, e.g., autonomous driving, robotics, and edge computing.Computing processors driven by electronics have evolved dramatically over the past decade, from general-purpose central processing units (CPUs) 1 to custom computing platforms, e.g., GPUs 2 , FPGAs 3 , and ASICs 4,5 , to meet the ubiquitously increasing demand of computing resources. The progress of these silicon computing hardware platforms has greatly contributed to the resurgence of artificial intelligence (AI) by allowing the training of larger-scale and more complicated models 6,7 . We have witnessed the extensive applications of various neural computing architectures, e.g., convolutional neural networks (CNNs) 2,7 , recurrent neural networks (RNNs) 8 , spiking neural networks (SNNs) 9 , and reservoir computing (RC) 10 , in a broad range of fields. However, electronic hardware implementations have reached unsustainable performance growth as the exponential scaling of electr...

Holography encodes the three dimensional (3D) information of a sample in the form of an intensity-only recording. However, to decode the original sample image from its hologram(s), auto-focusing and phase-recovery are needed, which are in general cumbersome and time-consuming to digitally perform. Here we demonstrate a convolutional neural network (CNN) based approach that simultaneously performs auto-focusing and phase-recovery to significantly extend the depth-of-field (DOF) in holographic image reconstruction. For this, a CNN is trained by using pairs of randomly de-focused back-propagated holograms and their corresponding in-focus phase-recovered images. After this training phase, the CNN takes a single back-propagated hologram of a 3D sample as input to rapidly achieve phase-recovery and reconstruct an in focus image of the sample over a significantly extended DOF. This deep learning based DOF extension method is non-iterative, and significantly improves the algorithm timecomplexity of holographic image reconstruction from O(nm) to O(1), where n refers to the number of individual object points or particles within the sample volume, and m represents the focusing search space within which each object point or particle needs to be individually focused. These results highlight some of the unique opportunities created by data-enabled statistical image reconstruction methods powered by machine learning, and we believe that the presented approach can be broadly applicable to computationally extend the DOF of other imaging modalities.

Figure 1: 3D hyperspectral (HS) images reconstructed from a single spatial-spectral encoded 2D projection for an outdoor scene under sunlight environment. Left-most: The sensor image captured by our camera prototype. Second-by-left: Recovered high-spatial resolution 3D HS images with color indicating 31 spectral bands from 420nm to 720nm. Third and fourth: Closeup of recovered 520nm and 650nm spectral bands images. Right-most: The synthetic RGB image from the reconstructed HS images. AbstractThis paper proposes a novel compressive hyperspectral (HS) imaging approach that allows for high-resolution HS images to be captured in a single image. The proposed architecture comprises three key components: spatial-spectral encoded optical camera design, over-complete HS dictionary learning and sparse-constraint computational reconstruction. Our spatial-spectral encoded sampling scheme provides a higher degree of randomness in the measured projections than previous compressive HS imaging approaches; and a robust nonlinear sparse reconstruction method is employed to recover the HS images from the coded projection with higher performance. To exploit the sparsity constraint on the nature HS images for computational reconstruction, an over-complete HS dictionary is learned to represent the HS images in a sparser way than previous representations. We validate the proposed approach on both synthetic and real captured data, and show successful recovery of HS images for both indoor and outdoor scenes. In addition, we demonstrate other applications for the over-complete HS dictionary and sparse coding techniques, including 3D HS images compression and denoising.

No abstract

No abstract

No abstract

Calcium imaging is inherently susceptible to detection noise especially when imaging with high frame rate or under low excitation dosage. We developed DeepCAD, a selfsupervised learning method for spatiotemporal enhancement of calcium imaging without requiring any high signal-to-noise ratio (SNR) observations. Using this method, detection noise can be effectively suppressed and the imaging SNR can be improved more than tenfold, which massively improves the accuracy of neuron extraction and spike inference and facilitate the functional analysis of neural circuits.Calcium imaging enables parallel recordings of large neuronal ensembles in living animals [1][2][3][4] and offers a new possibility for deciphering information propagation, integration, and computation in neural circuits 5 . To obtain accurate neuron extraction and spike inference for downstream neuroscience analysis, high-SNR calcium imaging is desired. However, due to the paucity of fluorescence photons caused by low peak accumulations and fast dynamics of in vivo calcium transients 6,7 , calcium imaging is easy to be contaminated by detection noise (i.e. photon shot noise and electronic noise), especially in functional imaging where high temporal resolution is particularly important for analyzing neural activities 8 .To capture sufficient fluorescence photons for high-SNR calcium imaging, the most direct way is to use high excitation dosage, but concurrent photobleaching, phototoxicity 9,10 , and tissue heating 11 are detrimental for sample health and photosensitive biological processes, which limits the maximal excitation power for long-term in vivo imaging 12 . More effective .