Deep Learning with Multimodal Representation for Pancancer Prognosis Prediction

Abstract

Estimating the future course of cancer is invaluable to physicians; however, current clinical methods fail to effectively use the vast amount of multimodal data that is available for cancer patients.To tackle this problem, we constructed a deep neural network based model to predict the survival of patients for 20 different cancer types using gene expressions, microRNA data, clinical data and histopathology whole slide images (WSIs). We developed an unsupervised encoder to compress these four data modalities into a single feature vector for each patient, handling missing data through a resilient, multimodal dropout method. Encoding methods were tailored to each data type -using deep highway networks to extract features from genomic and clinical data, and convolutional neural networks extract features from pathology images. We then used these feature encodings trained on pancancer data to predict pancancer and single cancer survival data, achieving a C-index of 0.784 overall.This work shows that it is possible to build a pancancer model for prognosis that also predicts prognosis in single cancer sites. Furthermore, our model handles multiple data modalities, efficiently analyzes WSIs, and summarizes patient details flexibly into an unsupervised, informative profile. We thus present a powerful automated tool to accurately determine prognosis, a key step towards personalized treatment for cancer patients.

Integrating multi-omics data through deep learning for accurate cancer prognosis prediction

“…Table 3 also showed the results of the DCAP using multi-omics data at last column. In comparison, the C-index obtained by three traditional methods (general Cox model, Cox model with lasso regularization, and Cox model with elastic net) achieved C-index values between 0.565 and 0.569, which were much lower than those obtained by some advanced methods in recent studies (Cox_DL, Cox_transfer, Cox_TRACE, Cox_cCMTL) (Cheerla and Gevaert, 2019;Wang, et al, 2017). These advanced methods obtained C-index values between 0.605 and 0.632, with an average of 0.620.…”

“…The solid lines were drawn by DCAP and the dotted lines (grey) were drawn by DCAP-kmeans. (Wang, et al, 2017) b : The results reported in reference (Cheerla and Gevaert, 2019) c : DCAP using only mRNA Figure. 3 The C-index values obtained by DCAP models in cross-cancer prognosis prediction. The DCAP models trained on one cancer are used to predict prognosis on another cancer.…”

Data-Driven Modeling of Pregnancy-Related Complications

“…For this, it combines models that excel at processing each of these modalities into a single model. For example, an integration of histological images and gene expression data has been shown to improve pan-cancer prognosis [190]. At the same time, pregnancy outcomes are highly interrelated and may point towards different phenotypes with similar pathologies.…”

A meta-learning approach for genomic survival analysis

“…The problem of small data size may be a limiting factor in many biomedical analyses, especially when studies are conducted with data that is expensive to produce, or in the case of multi-modal data (Cheerla and Gevaert, 2019). Our work shows the promise of meta-learning for biomedical applications to alleviate the problem of limited data.…”

“…For example, autoencoder architectures have been employed to extract features from genomic data for liver cancer prognosis prediction (Chaudhary et al, 2018). The Cox model has also been integrated in a neural network setting to allow greater modeling flexibility (Cheerla and Gevaert (2019); Ching et al (2018); Luck et al (2017); Yousefi et al (2017).…”