Fast intraoperative histology-based diagnosis of gliomas with third harmonic generation microscopy and deep learning

Abstract

Management of gliomas requires an invasive treatment strategy, including extensive surgical resection. The objective of the neurosurgeon is to maximize tumor removal while preserving healthy brain tissue. However, the lack of a clear tumor boundary hampers the neurosurgeon’s ability to accurately detect and resect infiltrating tumor tissue. Nonlinear multiphoton microscopy, in particular higher harmonic generation, enables label-free imaging of excised brain tissue, revealing histological hallmarks within seconds. Here, we demonstrate a real-time deep learning-based pipeline for automated glioma image analysis, matching video-rate image acquisition. We used a custom noise detection scheme, and a fully-convolutional classification network, to achieve on average 79% binary accuracy, 0.77 AUC and 0.83 mean average precision compared to the consensus of three pathologists, on a preliminary dataset. We conclude that the combination of real-time imaging and image analysis shows great potential for intraoperative assessment of brain tissue during tumor surgery.

Automated glioma multiclass tumor classification

“…There are strong hopes that automatic grading can be useful during intraoperative settings and to validate immunotherapies. 16 Future works include the use generative adversarial networks to augment the data seamlessly.…”

“…The purpose of this approach follows the hypothesis that automatic grade and subtyping from histological slides can be an interoperative step during surgery as well as other noninvasive therapy planning. 16 In addition, researchers expect that neuropathologists and machine learning models would make different sorts of categorization errors, and the aggregate assessment of a hybrid pathologist/machine learning model will be superior to either human or machine assessment alone. 17 Data used for the model training were digital images of tissue samples stained to reveal human leukocyte antigen (HLA-DR) staining on human glioma tissue microarrays (TMAs).…”

Compact portable higher harmonic generation microscopy for the real time assessment of fresh thyroid tissue

“…Here, we add third harmonic generation microscopy to the SHG and MPEF imaging modalities as THG has been successful in the imaging of tumor cells and nuclei in unprocessed tissue. THG provides contrast through the nonlinear susceptibility or index of refraction change at interfaces and inhomogeneities (20-22) and has been shown to have the capacity to generate high quality histopathological images of freshly excised tissue in less than one minute, in case of brain, breast, skin and lung tumor tissue (10,11,(14)(15)(16)(25)(26)(27). The combination of THG, SHG and MPEF, denoted here as HHGM, therefore seems to be a promising tool for real-time assessment of freshly excised tissue during surgery and could be promising for thyroid tissue as well.…”

Compact portable higher harmonic generation microscopy for the real time assessment of unprocessed thyroid tissue

“…Here, we add third harmonic generation microscopy to the SHG and MPEF imaging modalities as THG has been successful in the imaging of tumor cells and nuclei in unprocessed tissue. THG provides contrast through the nonlinear susceptibility or index of refraction change at interfaces and inhomogeneities (20)(21)(22) and has been shown to have the capacity to generate high quality histopathological images of freshly excised tissue in less than one minute, in case of brain, breast, skin and lung tumor tissue (10,11,(14)(15)(16)(25)(26)(27). The combination of THG, SHG and MPEF, denoted here as HHGM, therefore seems to be a promising tool for real-time assessment of freshly excised tissue during surgery and could be promising for thyroid tissue as well.…”