Recurrent neural network-based volumetric fluorescence microscopy

Abstract

Volumetric imaging of samples using fluorescence microscopy plays an important role in various fields including physical, medical and life sciences. Here we report a deep learning-based volumetric image inference framework that uses 2D images that are sparsely captured by a standard wide-field fluorescence microscope at arbitrary axial positions within the sample volume. Through a recurrent convolutional neural network, which we term as Recurrent-MZ, 2D fluorescence information from a few axial planes within the sample is explicitly incorporated to digitally reconstruct the sample volume over an extended depth-of-field. Using experiments on C. elegans and nanobead samples, Recurrent-MZ is demonstrated to significantly increase the depth-of-field of a 63×/1.4NA objective lens, also providing a 30-fold reduction in the number of axial scans required to image the same sample volume. We further illustrated the generalization of this recurrent network for 3D imaging by showing its resilience to varying imaging conditions, including e.g., different sequences of input images, covering various axial permutations and unknown axial positioning errors. We also demonstrated wide-field to confocal cross-modality image transformations using Recurrent-MZ framework and performed 3D image reconstruction of a sample using a few wide-field 2D fluorescence images as input, matching confocal microscopy images of the same sample volume. Recurrent-MZ demonstrates the first application of recurrent neural networks in microscopic image reconstruction and provides a flexible and rapid volumetric imaging framework, overcoming the limitations of current 3D scanning microscopy tools.

Deep-3D Microscope: 3D volumetric microscopy of thick scattering samples using a wide-field microscope and machine learning

“…We can see that the model trained for the original data (without downsampling) works well for the 2x downsampling, and is slightly degraded for 4x downsampled z-stacks. The spacing of 4x downsampled images in z-axis is comparable to the spacing Huang et al demonstrated [33], but we show good results on samples with much more complicated structures and unwanted backgrounds. The degradation is mainly caused by the gap between captured input layers and the interpolated layers.…”

“…Over the last few years researchers have developed early examples of deep convolutional neural networks to enhance axial resolution and imaging contrast of wide-field images [29][30][31][32][33]. Specifically, Zhang et al [29] first successfully transformed wide-field images to background reduced Structured Illumination Microscopy (SIM) images using a 2D neural network.…”

“…Although they subsequently demonstrated more complicated mice whole-brain images reconstruction in [31], the 2D convolutional structure confined the application of the network to physically sectioned tissue samples. Wu et al proposed two more sophisticated Generative Adversarial Network (GAN) based deep networks to predict confocal images from a single in-focus wide-field input [28] and sparse wide-field scans [33], respectively. However, they only testified this idea using 2.4 microns thickness BPAEC sample or a limited z-span of C-Elegans sample (about 4 microns).…”

Mirror-enhanced scanning light-field microscopy for long-term high-speed 3D imaging with isotropic resolution

“…However, current microscopy techniques usually image a certain plane at one time, with three-dimensional (3D) imaging obtained through movements of the focal plane relative to the specimen, such as confocal 1 , 2 , structured illumination 3 – 6 , and light-sheet microscopy 7 – 9 . To address this problem, a few emerging imaging techniques 10 – 15 aimed at simultaneous 3D imaging have been developed recently. Among them, light-field microscopy (LFM) 16 – 18 provides an elegant compact solution by capturing the excited volume simultaneously in a tomographic manner.…”

Deep Learning-powered Bessel-beam Multi-parametric Photoacoustic Microscopy

“…However, the previous studies were focused on structural imaging, leaving its utility in functional and quantitative imaging unexplored. In addition to the Bessel-beam excitation, deep learning has also been used to address the out-of-focus issue associated with the Gaussian beam [15], [16]. In particular, conditional generative adversarial networks (cGAN) have been successfully applied to biomedical image analysis [17]- [19].…”