Deep Learning on Multimodal Chemical and Whole Slide Imaging Data for Predicting Prostate Cancer Directly from Tissue Images

Abstract

Prostate cancer is one of the most common cancers globally and is the second most common cancer in the male population in the US. Here we develop a study based on correlating the hematoxylin and eosin (H&E)-stained biopsy data with MALDI mass-spectrometric imaging data of the corresponding tissue to determine the cancerous regions and their unique chemical signatures and variations of the predicted regions with original pathological annotations. We obtain features from high-resolution optical micrographs of whole slide H&E stained data through deep learning and spatially register them with mass spectrometry imaging (MSI) data to correlate the chemical signature with the tissue anatomy of the data. We then use the learned correlation to predict prostate cancer from observed H&E images using trained coregistered MSI data. This multimodal approach can predict cancerous regions with ∼80% accuracy, which indicates a correlation between optical H&E features and chemical information found in MSI. We show that such paired multimodal data can be used for training feature extraction networks on H&E data which bypasses the need to acquire expensive MSI data and eliminates the need for manual annotation saving valuable time. Two chemical biomarkers were also found to be predicting the ground truth cancerous regions. This study shows promise in generating improved patient treatment trajectories by predicting prostate cancer directly from readily available H&E-stained biopsy images aided by coregistered MSI data.

Deep MALDI-MS Spatial ‘Omics guided by Quantum Cascade Laser Mid-infrared Imaging Microscopy

“…In addition to these challenges, method development and validation in MSI have generally been hampered by the lack of reliable analytical ground truths for segmentation and molecular identities 22, 23 , as the spatial and molecular composition of investigated tissues is typically unknown. To this end, synthetic datasets 2 , expert crowdsourcing 22 single-cell fluorescence 24 , or histopathology annotations 25 have been proposed as ground truths. To address this key challenge in MSI method development and validation, we propose that genetic mouse models with defined alterations in metabolism be used as qualitative ground truth: In case of QCL-IRI-guided MSI workflows we employed arylsulfatase A-deficient (ARSA-/-) mice, a model of human metachromatic leukodystrophy (MLD).…”