Unsupervised content-preserving transformation for optical microscopy

Abstract

The advent of deep learning and the open access to a substantial collection of imaging data provide a potential solution to computational image transformation, which is gradually changing the landscape of optical imaging and biomedical research. However, current deep-learning implementations usually operate in a supervised manner, and the reliance on a laborious and error-prone data annotation procedure remains a barrier towards more general applicability. Here, we propose an unsupervised image transformation enlightened by cycle-consistent generative adversarial networks (cycleGANs) to facilitate the utilization of deep learning in optical microscopy. By incorporating the saliency constraint into cycleGAN, the unsupervised approach, dubbed as content-preserving cycleGAN (c 2 GAN), can learn the mapping between two image domains and avoid the misalignment of salient objects without paired training data. We demonstrate several image transformation tasks such as fluorescence image restoration, whole-slide histological coloration, and virtual fluorescent labeling. Quantitative evaluations prove that c 2 GAN achieves robust and high-fidelity image transformation across different imaging modalities and various data configurations. We anticipate that our framework will encourage a paradigm shift in training neural networks and democratize deep learning algorithms for optical society.Deep learning 1 has made great progress in computational imaging and image interpretation 2,3 . As a data-driven methodology, deep neural networks with high model capacity can theoretically approximate arbitrary function that represents the mapping from the input domain to the output domai 4,5 . Given images as the inputs in high-dimensional space, the applications of deep learning in optical microscopy can be divided into two categories according to the forms of the outputs.

Self-supervised classification of subcellular morphometric phenotypes reveals extracellular matrix-specific morphological responses

“…To overcome this, amalgamation of images from different modality such as phase contrast or quantitative phase imaging 45 , preferably on the same microscope platform, may be particularly useful 46 . Additionally, it may be instructive to compare or integrate our approach with recent studies making use of deep learning to perform artificial fluorescent labelling 47 .…”

Prostate cancer histopathology with label-free multispectral deep UV microscopy quantifies phenotypes of tumor grade and aggressiveness

“…The Adam optimizer was used to optimize network parameters (61). The initial learning rate is 0.0002, which decays linearly every 50 iterations with a rate of 0.99.…”

“…To translate the label-free UV images into virtual H&E images, we apply a recently developed unsupervised content-preserving transformation for optical microscopy (UTOM) deep neural network (61). UTOM adapts the general framework of cycle-consistent generative adversarial networks (Cycle-GAN) which can transform images from one domain into another without requiring pixel-level paired data.…”

“…In UTOM, a forward and backward GAN are trained simultaneously to learn a pair of opposite mappings between the UV and H&E image domains, as shown in Fig 6a . In this process, a cycle-consistency loss constrain, and a pair of saliency constraints are imposed to correct for mapping direction, which avoids distortions (Fig. 6a) (61).…”

High-throughput, label-free and slide-free histological imaging by computational microscopy and unsupervised learning

“…3,5). We believe that integrating weakly-/semi-supervised data or introducing a saliency constraint 41 would help to address this problem, further improving the accuracy of Deep-CHAMP images. Virtual staining through unsupervised learning should be systematically investigated in the future to enable a faithful conversion, which however, is beyond the scope of this study.…”