Deep learning-based image processing in optical microscopy

Abstract

Optical microscopy has emerged as a key driver of fundamental research since it provides the ability to probe into imperceptible structures in the biomedical world. For the detailed investigation of samples, a high-resolution image with enhanced contrast and minimal damage is preferred. To achieve this, an automated image analysis method is preferable over manual analysis in terms of both speed of acquisition and reduced error accumulation. In this regard, deep learning (DL)-based image processing can be highly beneficial. The review summarises and critiques the use of DL in image processing for the data collected using various optical microscopic techniques. In tandem with optical microscopy, DL has already found applications in various problems related to image classification and segmentation. It has also performed well in enhancing image resolution in smartphone-based microscopy, which in turn enablse crucial medical assistance in remote places. Graphical abstract

Semi-supervised machine learning workflow for analysis of nanowire morphologies from transmission electron microscopy images

“…Therefore, machine learning (ML) has become a valuable tool that can replace time-consuming and subjective manual interpretation of microscopy images with automated, objective, and fast analysis. [4][5][6][7][8] Recent advances in deep learning have led to a surge of applications in electron microscopy image analysis for a diverse set of tasks in two main categories: discriminative and generative. Discriminative tasks are tasks like morphology/phase classication, [9][10][11][12] particle/defect detection, [13][14][15][16] image quality assessment, [17][18][19] and segmentation [20][21][22][23][24][25] where the objective is quantied by how well the model can distinguish (1) between images or (2) between objects and their background.…”

“…35 Transfer learning involves leveraging the knowledge of a model previously trained using large training datasets to create a new model for another related task. For example, a model that has been trained on ImageNet, 36 a large dataset of 1.2 million photographic images of macroscopic objects, can be transferred to learn how to analyze images in another more specic domain [e.g., medical image analysis 37,38 and electron microscopy image analysis in material science [4][5][6][7][8] ]. The success of transfer learning in the eld of image analysis has paved the way for accessibility to pretrained image learning models for the general public without requiring large computational resources or big data to train from scratch.…”

“…Therefore, machine learning (ML) has become a valuable tool that can replace time-consuming and subjective manual interpretation of microscopy images with automated, objective, and fast analysis. 4–8…”

Utilizing deep learning for dermal matrix quality assessment on in vivo line‐field confocal optical coherence tomography images

“…Moreover, as hundreds of images can be analyzed to generate significant results, this process is time‐consuming 19 . For several years, deep learning‐based image analysis became highly beneficial, both in terms of speed of analysis and reduced error accumulation 20 . Such methods are used in the medical field for image segmentation, prediction, or classification for several years, thus helping in the diagnosis of clinicians 21 .…”

Pixel-level multimodal fusion deep networks for predicting subcellular organelle localization from label-free live-cell imaging

“…The learning-enhanced cell optical image-analysis model is capable of acquiring the texture details from low-level source images and achieve higher resolution improvement for the label-free cell optical-imaging techniques ( Chen et al, 2016 ; Lee et al, 2020 ; Ullah et al, 2021 ; Ullah et al, 2022 ). The deep-learning pipeline of cell optical microscopy imaging can extract complex data representation in a hierarchical way, which is helpful to find hidden cell structures from the microscope images, such as the size of a single cell, the number of cells in a given area, the thickness of the cell wall, the spatial distribution between cells, and subcellular components and their densities ( Boslaugh and Watters, 2008 ; Donovan-Maiye et al, 2018 ; Falk et al, 2019 ; Manifold et al, 2019 ; Rezatofighi et al, 2019 ; Yao et al, 2019 ; Zhang et al, 2019 ; Lee et al, 2020 ; Voronin et al, 2020 ; Zhang et al, 2020 ; Chen et al, 2021a ; Gomariz et al, 2021 ; Manifold et al, 2021 ; Wang et al, 2022b ; Islam et al, 2022 ; Kim et al, 2022 ; Melanthota et al, 2022 ; Rahman et al, 2022 ; Ullah et al, 2022 ; Witmer and Bhanu, 2022 ).…”