Resolution Enhancement with a Task-Assisted GAN to Guide Optical Nanoscopy Image Analysis and Acquisition

Abstract

We introduce a deep learning model for resolution enhancement and prediction of super-resolved biological structures, which is based on a Generative Adversarial Network (GAN) assisted by a complementary segmentation task. It is applied to predict biological nanostructures from diffraction-limited images and to guide microscopists for quantitative fixed- and live-cell STimulated Emission Depletion (STED) microscopy. More specifically, we show that the use of a complementary segmentation task improves the accuracy of the predicted nanostructures over state-of-the art resolution enhancement generative approaches, allowing quantitative analysis of the sub-diffraction structures in the resulting generated images.

Development of AI-assisted microscopy frameworks through realistic simulation in pySTED

“…A recent trend in microscopy focuses on the implementation of data-driven microscopy systems. For example, systems are built to automatically select informative regions or improve the quality of the acquired images [60,61]. The development and validation of such data-driven systems could be achieved with pySTED.…”

Spatial resolution enhancement in photon-starved STED imaging using deep learning-based fluorescence lifetime analysis

“…S6 †). Although GAN-based approaches have been used to transform the confocal images to match the resolution of STED images, 33,34 to the best of our knowledge, this is the first report that the GAN is applied to improve the spatial resolution of STED imaging based on the fluorescence temporal dynamics.…”