Accurate phase retrieval of complex 3D point spread functions with deep residual neural networks

Möckl, Leonhard; Petrov, Petar N.; Moerner, W. E.

doi:10.1063/1.5125252

Cited by 39 publications

(35 citation statements)

References 30 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It was our goal to combine a compact architecture to enable rapid, real-time PR with large Zernike coefficient amplitude ranges. [102] Additionally, we wanted to ensure that the approach is purely based on the NN, i.e. that there is no requirement for refinement of the predictions of the NN.…”

Section: Psf Analysis For Extraction Of Molecular and Imaging Parametersmentioning

confidence: 99%

Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]

Möckl¹,

Roy²,

Moerner³

2020

Biomed. Opt. Express

Self Cite

View full text Add to dashboard Cite

Deep learning-based data analysis methods have gained considerable attention in all fields of science over the last decade. In recent years, this trend has reached the single-molecule community. In this review, we will survey significant contributions of the application of deep learning in single-molecule imaging experiments. Additionally, we will describe the historical events that led to the development of modern deep learning methods, summarize the fundamental concepts of deep learning, and highlight the importance of proper data composition for accurate, unbiased results.

show abstract

Section: Psf Analysis For Extraction Of Molecular and Imaging Parametersmentioning

confidence: 99%

Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]

Möckl¹,

Roy²,

Moerner³

2020

Biomed. Opt. Express

Self Cite

View full text Add to dashboard Cite

show abstract

“…The emergence of deep learning has revolutionized the field of image processing. In particular, methods have been proposed for the deblurring of video sequences (Wieschollek et al 2017), for the rapid estimation of PSFs from images (Möckl et al 2019), and for the modeling of simple PSFs (Herbel et al 2018) for large-scale surveys. It is evident that methods that make use of many frames to produce a single deconvolved frame make much better use of the collected photons and should always be preferred over lucky imaging techniques.…”

Section: Introductionmentioning

confidence: 99%

Learning to do multiframe wavefront sensing unsupervised: Applications to blind deconvolution

Ramos

Olspert

2021

A&A

View full text Add to dashboard Cite

Context. Observations from ground-based telescopes are severely perturbed by the presence of the Earth’s atmosphere. The use of adaptive optics techniques has allowed us to partly overcome this limitation. However, image-selection or post-facto image-reconstruction methods applied to bursts of short-exposure images are routinely needed to reach the diffraction limit. Deep learning has recently been proposed as an efficient way to accelerate these image reconstructions. Currently, these deep neural networks are trained with supervision, meaning that either standard deconvolution algorithms need to be applied a priori or complex simulations of the solar magneto-convection need to be carried out to generate the training sets. Aims. Our aim here is to propose a general unsupervised training scheme that allows multiframe blind deconvolution deep learning systems to be trained with observations only. The approach can be applied for the correction of point-like as well as extended objects. Methods. Leveraging the linear image formation theory and a probabilistic approach to the blind deconvolution problem produces a physically motivated loss function. Optimization of this loss function allows end-to-end training of a machine learning model composed of three neural networks. Results. As examples, we apply this procedure to the deconvolution of stellar data from the FastCam instrument and to solar extended data from the Swedish Solar Telescope. The analysis demonstrates that the proposed neural model can be successfully trained without supervision using observations only. It provides estimations of the instantaneous wavefronts, from which a corrected image can be found using standard deconvolution techniques. The network model is roughly three orders of magnitude faster than applying standard deconvolution based on optimization and shows potential to be used on real-time at the telescope.

show abstract

“…Over the last years, deep learning-based approaches using convolutional neural networks (CNNs) have proven to be powerful and computationally efficient for image-based classification and regression tasks for microscopy images [18,19]. Recently, several studies demonstrated that deep learning-based phase retrieval can produce accurate results at fast processing speeds [20][21][22][23][24][25], however they fall short regarding their practical applicability. Some of these approaches [22][23][24] used purely simulated synthetic data, where generalizability to real microscopy images is unclear.…”

Section: Introductionmentioning

confidence: 99%

“…Recently, several studies demonstrated that deep learning-based phase retrieval can produce accurate results at fast processing speeds [20][21][22][23][24][25], however they fall short regarding their practical applicability. Some of these approaches [22][23][24] used purely simulated synthetic data, where generalizability to real microscopy images is unclear. Others focused on specific microscopy acquisition modes (such as using biplanar PSFs [20]) or microscopy setups that allow to collect large sets of experimental ground truth data for training and prediction [21,25], thus limiting this approach in practice.…”

Section: Introductionmentioning

confidence: 99%

Practical sensorless aberration estimation for 3D microscopy with deep learning

et al. 2020

View full text Add to dashboard Cite

Estimation of optical aberrations from volumetric intensity images is a key step in sensorless adaptive optics for 3D microscopy. Recent approaches based on deep learning promise accurate results at fast processing speeds. However, collecting ground truth microscopy data for training the network is typically very difficult or even impossible thereby limiting this approach in practice. Here, we demonstrate that neural networks trained only on simulated data yield accurate predictions for real experimental images. We validate our approach on simulated and experimental datasets acquired with two different microscopy modalities and also compare the results to non-learned methods. Additionally, we study the predictability of individual aberrations with respect to their data requirements and find that the symmetry of the wavefront plays a crucial role. Finally, we make our implementation freely available as open source software in Python.

show abstract

Accurate phase retrieval of complex 3D point spread functions with deep residual neural networks

Cited by 39 publications

References 30 publications

Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]

Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]

Learning to do multiframe wavefront sensing unsupervised: Applications to blind deconvolution

Practical sensorless aberration estimation for 3D microscopy with deep learning

Contact Info

Product

Resources

About