On the use of deep learning for computational imaging

Barbastathis, George; Ozcan, Aydogan; Situ, Guohai

doi:10.1364/optica.6.000921

Cited by 595 publications

(252 citation statements)

References 268 publications

Supporting

Mentioning

225

Contrasting

Unclassified

Order By: Relevance

“…Ghost imaging (GI) is an imaging technique that generates the image of an object by calculating the second-order correlation function between two light beams [1][2][3][4][5][6]. Ghost imaging has been widely researched in recent years [7][8][9][10][11][12][13][14][15][16]; it has important applications in many fields such as cryptography [17,18], lidar [19,20], medical imaging [21,22], micro object imaging [23,24], three-dimensional imaging [25][26][27][28] and single-pixel imaging [29][30][31][32][33]. In many practical scenes, GI is subject to interference from the transmission medium and from optical background noise.…”

Section: Introductionmentioning

confidence: 99%

Instant ghost imaging: improving robustness for ghost imaging subject to optical background noise

Yang

Zhang

et al. 2020

OSA Continuum

View full text Add to dashboard Cite

Ghost imaging (GI) is an imaging technique that uses the second-order correlation between two light beams to obtain the image of an object. However, standard GI is affected by optical background noise, which reduces its practical use. We investigated the robustness of an instant ghost imaging (IGI) algorithm against optical background noise and compare it with the conventional GI algorithm. Our results show that IGI is extremely resistant to spatiotemporally varying optical background noise that can change over a large range. When the noise is large in relation to the signal, IGI will still perform well in conditions that prevent the conventional GI algorithm from generating an image because IGI uses signal differences for imaging. Signal differences are intrinsically resistant to common noise modes, so the IGI algorithm is strongly robust against noise. This research is of great significance for the practical application of GI.

show abstract

Section: Introductionmentioning

confidence: 99%

Instant ghost imaging: improving robustness for ghost imaging subject to optical background noise

Yang

Zhang

et al. 2020

OSA Continuum

View full text Add to dashboard Cite

show abstract

“…The past decade has seen an explosion in the use of deep learning algorithms [13][14][15][16] across all areas of science. It is thus natural to consider whether temporal resolution can be improved using deep learning techniques.…”

Section: Introductionmentioning

confidence: 99%

Learned SPARCOM: Unfolded Deep Super-Resolution Microscopy

Dardikman-Yoffe

Eldar

2020

Preprint

View full text Add to dashboard Cite

The use of photo-activated fluorescent molecules to create long sequences of low emitter-density diffraction-limited images enables high-precision emitter localization. However, this is achieved at the cost of lengthy imaging times, limiting temporal resolution. In recent years, a variety of approaches have been suggested to reduce imaging times, ranging from classical optimization and statistical algorithms to deep learning methods. Classical methods often rely on prior knowledge of the optical system and require heuristic adjustment of parameters or do not lead to good enough performance. Deep learning methods proposed to date tend to suffer from poor generalization ability outside the specific distribution they were trained on, and require learning of many parameters. They also tend to lead to black-box solutions that are hard to interpret. In this paper, we suggest combining a recent high-performing classical method, SPARCOM, with model-based deep learning, using the algorithm unfolding approach which relies on an iterative algorithm to design a compact neural network considering domain knowledge. We show that the resulting network, Learned SPARCOM (LSPARCOM), requires far fewer layers and parameters, and can be trained on a single field of view. Nonetheless it yields comparable or superior results to those obtained by SPARCOM with no heuristic parameter determination or explicit knowledge of the point spread function, and is able to generalize better than standard deep learning techniques. It even allows producing a high-quality reconstruction from as few as 25 frames. This is due to a significantly smaller network, which also contributes to fast performance -5× improvement in execution time relative to SPARCOM, and a full order of magnitudes improvement relative to a leading competing deep learning method (Deep-STORM) when implemented serially. Our results show that we can obtain super-resolution imaging from a small number of high emitter density frames without knowledge of the optical system and across different test sets. Thus, we believe LSPARCOM will find broad use in single molecule localization microscopy of biological structures, and pave the way to interpretable, efficient live-cell imaging in a broad range of settings.

show abstract

“…It is important to consider that any output from a deep learning super-resolution model is a prediction, is never 100% accurate, and is always highly dependent on proper correspondence between the training versus testing data 3,4,20,22 . Whether the level of accuracy of a given model for a given dataset is sufficient is ultimately dependent on whether the tolerance for error in the measurement being made is higher than the actual error.…”

Section: Discussionmentioning

confidence: 99%

Deep Learning-Based Point-Scanning Super-Resolution Imaging

Fang

Monroe²,

Novak

et al. 2019

Preprint

View full text Add to dashboard Cite

Point scanning imaging systems (e.g. scanning electron or laser scanning confocal microscopes) are perhaps the most widely used tools for high resolution cellular and tissue imaging. Like all other imaging modalities, the resolution, speed, sample preservation, and signal-to-noise ratio (SNR) of point scanning systems are difficult to optimize simultaneously. In particular, point scanning systems are uniquely constrained by an inverse relationship between imaging speed and pixel resolution. Here we show these limitations can be mitigated via the use of deep learning-based super-sampling of undersampled images acquired on a point-scanning system, which we termed point-scanning super-resolution (PSSR) imaging. Oversampled, high SNR ground truth images acquired on scanning electron or Airyscan laser scanning confocal microscopes were 'crappified' to generate semi-synthetic training data for PSSR models that were then used to restore real-world undersampled images. Remarkably, our EM PSSR model could restore undersampled images acquired with different optics, detectors, samples, or sample preparation methods in other labs. PSSR enabled previously unattainable 2 nm resolution images with our serial block face scanning electron microscope system. For fluorescence, we show that undersampled confocal images combined with a multiframe PSSR model trained on Airyscan timelapses facilitates Airyscan-equivalent spatial resolution and SNR with ~100x lower laser dose and 16x higher frame rates than corresponding high-resolution acquisitions. In conclusion, PSSR facilitates point-scanning image acquisition with otherwise unattainable resolution, speed, and sensitivity. Fig. 1 | Restoration of semi-synthetic and real-world EM testing data using PSSR model trained on semi-synthetically generated training pairs. a, Overview of the general workflow.Training pairs were semi-synthetically created by applying a degrading function to the HR images taken from a scanning electron microscope in transmission mode (tSEM) to generate LR counterparts (left column). Semi-synthetic pairs were used as training data through a dynamic ResNet-based U-Net architecture (middle column). Real-world LR and HR image pairs were both manually acquired under a SEM (right column). The output from PSSR (LR-PSSR) when LR is served as input is then compared to HR to evaluate the performance of our trained model. b, Restoration performance on semi-synthetic testing pairs from tSEM. Shown is the same field of view of a representative bouton region from the synthetically created LR input with the pixel size of 8 nm (left column), a 16x bilinear upsampled image with 2 nm pixel size (second column), 16x PSSR upsampled result with 2 nm pixel size (third column) and the HR ground truth acquired at the microscope with the pixel size of 2 nm (fourth column). A close view of the same vesicle in each image is highlighted. The Peak-Signal-to-Noise-Ratio (PSNR) and the Structural Similarity (SSIM) quantification of the semi-synthetic testing sets are shown (right). c, Restor...

show abstract

On the use of deep learning for computational imaging

Cited by 595 publications

References 268 publications

Instant ghost imaging: improving robustness for ghost imaging subject to optical background noise

Instant ghost imaging: improving robustness for ghost imaging subject to optical background noise

Learned SPARCOM: Unfolded Deep Super-Resolution Microscopy

Deep Learning-Based Point-Scanning Super-Resolution Imaging

Contact Info

Product

Resources

About