DeepLoco: Fast 3D Localization Microscopy Using Neural Networks

Abstract

Single-molecule localization super-resolution microscopy (SMLM) techniques like STORM and PALM have transformed cellular microscopy by substantially increasing spatial resolution. In this paper we introduce a new algorithm for a critical part of the SMLM process: estimating the number and locations of the fluorophores in a single frame. Our algorithm can analyze a 20000-frame experimental 3D SMLM dataset in about one second -substantially faster than real-time and existing algorithms. Our approach is straightforward but very different from existing algorithms: we train a neural network to minimize the Bayes' risk under a generative model for single SMLM frames. The neural network maps a frame directly to a collection of fluorophore locations, which we compare to the ground truth using a novel loss function. While training the neural network takes several hours, it only has to be done once for a given experimental setup. After training, localizing fluorophores in new images is extremely fast -orders of magnitude faster than existing algorithms. Faster recovery opens the door to real-time calibration and accelerated acquisition, and future work could tackle more complicated optical systems and more realistic simulators.

High-throughput multicolor 3D localization in live cells by depth-encoding imaging flow cytometry

“…Using spectral separation of fluorescence emission, however, is not a fundamental requirement of this or other localization microscopy methods. The recent application of machine learning, namely neural networks (NNs), to localization microscopy has been shown by our group (Nehme et al, 2018(Nehme et al, , 2019 and others (Boyd et al, 2018;Ouyang et al, 2018) to be adept at extracting the positions of multiple closely-spaced emitters (>6 emitters/µm 2 ), and could represent one path to enable more complex samples. Favorably, NNs also have the advantage of rapid data processing, and have recently been applied to online analysis controlling cell sorting during IFC experiments in a custom instrument (Nitta et al, 2018).…”

Three dimensional nanoscopy of whole cells and tissues within situpoint spread function retrieval

“…The decoding method infers the location of a single molecule inside the biological specimen from the emission pattern detected on the camera. This inference process utilizes a wide range of well-established mathematical tools such as feature-based mapping [16,23], regression [24,25], and deep learning [26][27][28] to estimate the molecular position using a 3D PSF model, which describes the emission pattern of a single molecule with respect to its position within the specimen. It is, therefore, imperative to obtain an accurate model of the 3D PSF, which can reflect the complex biological and physical context constituting the path that the emitted photons travel through before being detected.…”

Deep learning achieves super-resolution in fluorescence microscopy

“…Recently, a number of studies have used deep learning-based approaches to advance optical microscopy techniques, including bright-field microscopy 12,13 , holographic phase microscopy [14][15][16] , and fluorescence microscopy. [17][18][19][20] Some of these earlier results on fluorescence microscopy have focused on faster image acquisition or inference for single molecule localization microscopy [17][18][19] , or resolution enhancement by learning a sample specific imaging process through simulations 20 . Unlike these contributions, our presented technique makes no prior assumptions regarding the imaging process, such as an approximate model of the point spread function [17][18][19][20] , and does not depend on an additional computational technique to generate the desired target images, using e.g., PSF-fitting to a sparse set of samples.…”

“…[17][18][19][20] Some of these earlier results on fluorescence microscopy have focused on faster image acquisition or inference for single molecule localization microscopy [17][18][19] , or resolution enhancement by learning a sample specific imaging process through simulations 20 . Unlike these contributions, our presented technique makes no prior assumptions regarding the imaging process, such as an approximate model of the point spread function [17][18][19][20] , and does not depend on an additional computational technique to generate the desired target images, using e.g., PSF-fitting to a sparse set of samples. [17][18][19] Rather than localizing specific filamentous structures of a sample, here we demonstrate the generalization of our approach by super-resolving various sub-cellular structures, such as nuclei, microtubules, Factin and mitochondria.…”

“…Unlike these contributions, our presented technique makes no prior assumptions regarding the imaging process, such as an approximate model of the point spread function [17][18][19][20] , and does not depend on an additional computational technique to generate the desired target images, using e.g., PSF-fitting to a sparse set of samples. [17][18][19] Rather than localizing specific filamentous structures of a sample, here we demonstrate the generalization of our approach by super-resolving various sub-cellular structures, such as nuclei, microtubules, Factin and mitochondria. We further demonstrate that the presented framework can be generalized to multiple microscopic imaging modalities, including cross-modality image transformations (e.g., confocal to STED) as we report in the Results section.…”