PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning

Abstract

Using a deep neural network, we demonstrate a digital staining technique, which we term PhaseStain, to transform the quantitative phase images (QPI) of label-free tissue sections into images that are equivalent to the brightfield microscopy images of the same samples that are histologically stained. Through pairs of image data (QPI and the corresponding brightfield images, acquired after staining), we train a generative adversarial network and demonstrate the effectiveness of this virtual-staining approach using sections of human skin, kidney, and liver tissue, matching the brightfield microscopy images of the same samples stained with Hematoxylin and Eosin, Jones’ stain, and Masson’s trichrome stain, respectively. This digital-staining framework may further strengthen various uses of label-free QPI techniques in pathology applications and biomedical research in general, by eliminating the need for histological staining, reducing sample preparation related costs and saving time. Our results provide a powerful example of some of the unique opportunities created by data-driven image transformations enabled by deep learning.

Revealing architectural order with quantitative label-free imaging and deep learning

“…We systematically evaluated how the dimensions and input channels affect the prediction accuracy. Compared to previous work that predict fluorescence images from single label-free contrast (33)(34)(35)(36), we show that higher prediction accuracy can be achieved by combining multiple label-free contrasts. Additionally, we demonstrated prediction of fluorescence images of tissue, while previous work has reported prediction of fluorescence images of cultured cells or brightfield images of histochemically-stained tissue (33)(34)(35)(36).…”

“…We have also reported novel deep learning models for efficient analysis of multi-dimensional 3D data we acquire. In contrast to other work on image translation that demonstrated 2D prediction (34)(35)(36), our 2.5D architecture achieves 3D prediction with apparently similar or superior accuracy as the 3D prediction reported in (33) (Pearson correlation coefficient in 3D for nuclei prediction from brightfield images: 0.87 v.s. 0.7 reported in (33)), while being computationally more efficient.…”

“…We find that higher prediction accuracy is achieved by combining multiple label-free contrasts, in contrast to previous work that predicts fluorescence images from single label-free contrast. Additionally, we demonstrated prediction of fluorescence images of tissue, while previous work has only reported prediction of fluorescence images of cultured cells or brightfield images of histochemically-stained tissue (35)(36)(37)40). Image translation results reported in (35)(36)(37) were limited to 2D structures, likely because they use even more complex architectures than 3D U-Net model reported in (40).…”

“…Phase-contrast and DIC were combined with adaptation of the inception model for in silico 2D labeling of nuclei, cell types, and cell state (35). Quantitative phase (36) and auto-fluorescence (37) were combined with generative adverserial networks for prediction of 2D histopathological stains. Diffraction tomography was combined with U-Net for measuring 3D shape of immunological synapse (38).…”

“…Machine learning models have recently enabled identification of structures in diverse label-free images, opening new opportunities for scalable analysis of label-free data. Some examples include: 3D U-Net model for predicting multiple organelles in cells from brightfield and DIC images (33); inception model for in silico 2D labeling of nuclei, cell types, and cell state (34) from phase contrast and DIC images; generative adversarial models for 2D prediction of histopathological stains from quantitative phase (35) and auto-fluorescence (36); 3D U-Net model for segmentation of immunological synapse from diffraction tomography (37); random forest classifiers for recognizing carcinoma in colon tissue from Raman scattering (38). Among the above approaches, 3D U-Net models that translate label-free image stacks to a fluorescence stacks (33) are attractive, because they predict both localization and expression of specific molecules in 3D.…”