Multi-Channel 3D Deep Feature Learning for Survival Time Prediction of Brain Tumor Patients Using Multi-Modal Neuroimages

Abstract

High-grade gliomas are the most aggressive malignant brain tumors. Accurate pre-operative prognosis for this cohort can lead to better treatment planning. Conventional survival prediction based on clinical information is subjective and could be inaccurate. Recent radiomics studies have shown better prognosis by using carefully-engineered image features from magnetic resonance images (MRI). However, feature engineering is usually time consuming, laborious and subjective. Most importantly, the engineered features cannot effectively encode other predictive but implicit information provided by multi-modal neuroimages. We propose a two-stage learning-based method to predict the overall survival (OS) time of high-grade gliomas patient. At the first stage, we adopt deep learning, a recently dominant technique of artificial intelligence, to automatically extract implicit and high-level features from multi-modal, multi-channel preoperative MRI such that the features are competent of predicting survival time. Specifically, we utilize not only contrast-enhanced T1 MRI, but also diffusion tensor imaging (DTI) and resting-state functional MRI (rs-fMRI), for computing multiple metric maps (including various diffusivity metric maps derived from DTI, and also the frequency-specific brain fluctuation amplitude maps and local functional connectivity anisotropy-related metric maps derived from rs-fMRI) from 68 high-grade glioma patients with different survival time. We propose a multi-channel architecture of 3D convolutional neural networks (CNNs) for deep learning upon those metric maps, from which high-level predictive features are extracted for each individual patch of these maps. At the second stage, those deeply learned features along with the pivotal limited demographic and tumor-related features (such as age, tumor size and histological type) are fed into a support vector machine (SVM) to generate the final prediction result (i.e., long or short overall survival time). The experimental results demonstrate that this multi-model, multi-channel deep survival prediction framework achieves an accuracy of 90.66%, outperforming all the competing methods. This study indicates highly demanded effectiveness on prognosis of deep learning technique in neuro-oncological applications for better individualized treatment planning towards precision medicine.

Deep learning radiomics can predict axillary lymph node status in early-stage breast cancer

“…Network feature extraction. Contrasting with hand-crafted and engineered features designed according to the previous medical experiences, DLR learnt the highthroughput image features in a supervised manner, which could make full use of all embedded information in US images 50,51 . The convolutional layers encoded the input rectangular ROIs and adaptively learnt the semantic features and the FC layers then selected the relevant features and reduced the features dimensions.…”

Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines

“…Seven of the early fusion studies compared the performance of their fusion models against single modality models ( Table 1). Six of these studies showed an improvement in performance when using fusion 25,26,28,29,31,33 , and the remaining one achieved the same performance but reduced standard deviation 27 , alluding to a model with better stability.…”

Brain Tumor Analysis Empowered with Deep Learning: A Review, Taxonomy, and Future Challenges

“…The 3D convolutional neural network consists of multi-channel metric maps that are used to extract the high-grade predictive features from the individual patch of these maps, and trains the network layers for prediction. In the second stage, Support Vector Machine (SVM) are used to classify tumor-related features such as age, histological type, and tumor size to predict the final (short or long) overall survival time of high-grade gliomas patients with 90.66% accuracy [136].…”