High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network

Abstract

We combine generative adversarial network (GAN) with light microscopy to achieve deep learning super-resolution under a large field of view (FOV). By appropriately adopting prior microscopy data in an adversarial training, the neural network can recover a high-resolution, accurate image of new specimen from its single low-resolution measurement. Its capacity has been broadly demonstrated via imaging various types of samples, such as USAF resolution target, human pathological slides, fluorescence-labelled fibroblast cells, and deep tissues in transgenic mouse brain, by both wide-field and light-sheet microscopes. The gigapixel, multi-color reconstruction of these samples verifies a successful GAN-based single image super-resolution procedure. We also propose an image degrading model to generate low resolution images for training, making our approach free from the complex image registration during training dataset preparation. After a welltrained network being created, this deep learning-based imaging approach is capable of recovering a large FOV (~95 mm 2 ), high-resolution (~1.7 μm) image at high speed (within 1 second), while not necessarily introducing any changes to the setup of existing microscopes.

Widefield light sheet microscopy using an Airy beam combined with deep-learning super-resolution

“…This is consistent with previous observations in conventional widefield epifluorescence microscopy [18]. In LSM, deep learning has been applied previously in one instance, however, the training was performed using artificially simulated low-resolution images from high-resolution data [27]. In microscopy, deep-learning has been typically used to surpass the diffraction limit to achieve super-resolution microscopy [28].…”

Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]

“…image restoration. [50] GANs consist of two NNs: The generator network creates images from a noise vector (i.e. a vector of random numbers) by spatial upsampling and 2D convolution.…”

Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy

“…deconvolution, it is possible to train the deep network using simulated image data [96], i.e., we can assume an accurate knowledge about the blurring kernel and simulate the low-resolution data from the high-resolution images to create the training image set. However, in most practical cases, this estimation will not yield satisfactory results.…”