Virtual Stain Transfer in Histology via Cascaded Deep Neural Networks

Abstract

Pathological diagnosis relies on the visual inspection of histologically stained thin tissue specimens, where different types of stains are applied to bring contrast to and highlight various desired histological features. However, the destructive histochemical staining procedures are usually irreversible, making it very difficult to obtain multiple stains on the same tissue section. Here, we demonstrate a virtual stain transfer framework via a cascaded deep neural network (C-DNN) to digitally transform hematoxylin and eosin (H&E) stained tissue images into other types of histological stains. Unlike a single neural network structure which only takes one stain type as input to digitally output images of another stain type, C-DNN first uses virtual staining to transform autofluorescence microscopy images into H&E and then performs stain transfer from H&E to the domain of the other stain in a cascaded manner. This cascaded structure in the training phase allows the model to directly exploit histochemically stained image data on both H&E and the target special stain of interest. This advantage alleviates the challenge of paired data acquisition and improves the image quality and color accuracy of the virtual stain transfer from H&E to another stain. We validated the superior performance of this C-DNN approach using kidney needle core biopsy tissue sections and successfully transferred the H&E-stained tissue images into virtual PAS (periodic acid-Schiff) stain. This method provides high-quality virtual images of special stains using existing, histochemically stained slides and creates new opportunities in digital pathology by performing highly accurate stain-to-stain transformations.

Deep learning-enabled virtual histological staining of biological samples

“…Various network structures have been reported for virtual staining, among which the generative adversarial network 77 (GAN) is one of the most commonly and widely used frameworks due to its strong representation capability 18,[20][21][22][23]31,32,[36][37][38]48,51,56,59,78,79 . Compared to non-GAN-based inference models, GANs can generate relatively higher resolution and perceptually more realistic images 13,14,59 .…”

“…However, in the standard GAN framework where the Generator is solely optimized by an adversarial loss, the resulting Generator only mimics the colors and patterns of the target images without learning the underlying correspondence between the input and the target images, resulting in severe hallucinations at the micro-scale 19 . To overcome this hallucination problem, various other pixel-wise loss functions, such as mean absolute error (MAE) 18,21,22,32,36,37,51,56,59 , mean square error (MSE) 18,79 , SSIM 31,82 , Huber loss 31 , reversed Huber loss 23 , and color distance metrics 56 are incorporated into the Generator loss terms (in addition to the Discriminator loss) to regularize the GAN training; these additional loss terms are calculated using the virtually generated images and their corresponding ground truth (histochemically stained images). Moreover, image regularization terms such as total variation 83 were also exploited in some works to eliminate or suppress different types of image artifacts created by the Generator 18,[20][21][22]31 .…”

“…One approach used to mitigate this limitation and generate paired images with different stain types involved multiple pre-trained virtual staining networks 21 , which, however, may also induce an unavoidable distribution shift between the target histological images and the output from the pre-trained networks. To overcome this dilemma, Yang et al demonstrated a cascaded neural network (C-DNN) 79 architecture, where a virtual staining network A was followed by a jointly optimized stain transfer network B as shown in Fig. 7b.…”

“…7b. By using two groups of paired label-free, histochemically stained, and virtually stained images, the C-DNN used structural loss terms like MAE directly on histochemically stained images from both the input and output domains to improve the quality of virtual stain-to-stain transformations 79 . Unlike supervised learning, the training of virtual staining networks using unsupervised learning schemes does not require cross-registered image pairs.…”

Label-Free Analysis of Organelle Interactions Using Organelle-Specific Phase Contrast Microscopy (OS-PCM)

“…Meanwhile, specificity assignment (automatic identification of various organelles) is also necessary. Among machine learning approaches, using deep learning to give specificity to label-free imaging is an attractive option since deep learning has achieved wide-ranging success in microscopy, including enabling super-resolution, digital staining of tissues, , image restoration, among other applications. , However, the majority of segmentation of label-free image are limited to the whole cell level for the purposes of counting, classification, dry mass measurement, etc. Thus, individual organelles are not resolved.…”