Multimodal fusion of imaging and genomics for lung cancer recurrence prediction

Abstract

Lung cancer has a high rate of recurrence in early-stage patients. Predicting the post-surgical recurrence in lung cancer patients has traditionally been approached using single modality information of genomics or radiology images. We investigate the potential of multimodal fusion for this task. By combining computed tomography (CT) images and genomics, we demonstrate improved prediction of recurrence using linear Cox proportional hazards models with elastic net regularization. We work on a recent non-small cell lung cancer (NSCLC) radiogenomics dataset of 130 patients and observe an increase in concordance-index values of up to 10%. Employing non-linear methods from the neural network literature, such as multi-layer perceptrons and visual-question answering fusion modules, did not improve performance consistently. This indicates the need for larger multimodal datasets and fusion techniques better adapted to this biological setting.

Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data

“…Multimodal research involving radiology has been predominantly correlative in nature [13,12]. Some have explored late-stage fusion approaches combining feature-based representations from radiology with similar pathology [17] or genomic features [18] to predict recurrence. While promising, these strategies rely on hand-crafted feature sets and simple multimodal classifiers that likely limit their ability to learn complex prognostic interactions between modalities and realize the full additive benefit of integrating diverse clinical modalities.…”