Real-time intraoperative diagnosis by deep neural network driven multiphoton virtual histology

Abstract

Recent advances in label-free virtual histology promise a new era for real-time molecular diagnosis in the operating room and during biopsy procedures. To take full advantage of the rich, multidimensional information provided by these technologies, reproducible and reliable computational tools that could facilitate the diagnosis are in great demand. In this study, we developed a deep-learning-based framework to recognize cancer versus normal human breast tissue from real-time label-free virtual histology images, with a tile-level AUC (area under receiver operating curve) of 95% and slide-level AUC of 100% on unseen samples. Furthermore, models trained on a high-quality laboratory-generated dataset can generalize to independent datasets acquired from a portable intraoperative version of the imaging technology with a physics-based adapted design. Classification activation maps and final feature visualization revealed discriminative patterns, such as tumor cells and tumor-associated vesicles, that are highly associated with cancer status. These results demonstrate that through the combination of real-time virtual histopathology and a deep-learning framework, accurate real-time diagnosis could be achieved in point-of-procedure clinical applications.

Intraoperative Label-Free Multimodal Nonlinear Optical Imaging for Point-of-Procedure Cancer Diagnostics

“…Most recently, with the development of artificial intelligence (AI), we have combined deep learning (DL) with our label-free multimodal NLOI techniques to enable real-time intraoperative virtual histology for cancer diagnosis [ 67 ]. The high dimensional multimodal image-based data sets generated by these NLOI systems provide a wealth of features, textures, patterns, and objects for AI/DL analysis.…”

“…This multimodal imaging therefore can represent both the microstructures and metabolic profiles of cells and tissues, and do this rapidly in real-time to capture cell and tissue dynamics. Our Biophotonics Imaging Laboratory has developed many label-free multimodal nonlinear optical imaging (NLOI) techniques for simultaneous multi-contrast imaging [57]- [67], including 2PF, 3PF, SHG, THG, and CARS. 2PF and 3PF signals give the distribution of FAD and NAD(P)H, respectively.…”

“…CARS at 2850 cm −1 and 3050 cm −1 correspond to the CH 2 and CH aromatic stretching, which indicates the relative lipid and protein content, respectively [ 57 ]–[ 59 ]. These combined nonlinear modalities can provide label-free molecular profiling of intact tissue specimens, which is becoming increasingly important for cancer diagnosis [ 57 ]–[ 67 ].…”

Roadmap on bio-nano-photonics

“…a relatively limited number of label-free optical signatures can be addressed by the development of multimodal imaging systems that can, within a single shot, excite and collect multiple co-registered optical signatures that can be directly utilized by AI/ML/DL algorithms for automated classification, disease detection, and diagnosis (figure 11) [62,64].…”

Label-Free Deep Profiling of the Tumor Microenvironment

“…The computational pipeline developed here allows for deep profiling of cells and EVs within the original microenvironmental context in a single-cell/EV, label- free, and multilevel fashion, and will advance investigations and our understanding of the spatial heterogeneity of the tumor microenvironment. In addition, due to the label-free nature and the streamlined analytical capacity, the proposed computational pipeline can potentially serve as a complementary real-time analysis tool for point-of-procedure diagnosis (34), providing new insights into disease and therapy monitoring at the point-of-care through metabolic mapping of the tumor microenvironment (16,35).…”