Survival Prediction of Brain Cancer with Incomplete Radiology, Pathology, Genomic, and Demographic Data

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

“…Radiology imaging supports medical decisions by providing visible image contrasts inside the human body with radiant energy, including MRI, CT, positron emission tomography (PET), fMRI and x-ray, etc. To embed the intensity standardized 2D or 3D radiology images into feature representations with learning-based encoders [16,24,[34][35][36]87] or conventional radiomics methods [24,34,35] or both [34,35], skull-stripping [38], affine registration [38], foreground extraction [39], lesion segmentation [20,34,35,38] were used correspondingly in some reviewed works to define the ROIs at first. And then, the images were resized or cropped to a smaller size for feature extraction.…”

“…And then, the images were resized or cropped to a smaller size for feature extraction. In order to further reduce the image dimension to fit computation memory but keep essential information, some works used 2D maximum intensity projection [16], and some works used the representative slices with maximum tumor diameters [20,35] to convert the 3D volume to 2D. The fMRI is another radiology modality used in some reviewed studies that investigated autism spectrum disorder (ASD) and Alzheimer's disease (AD) [40,41].…”

“…Most of the clinical data and lab test results in the reviewed works are structured data and can be converted to feature vectors easily. In preprocessing, categorical clinical features were usually converted through one-hot encoding, while the numerical features were standardized [16,34,35]. Specially, Cai et al [51] used soft one-hot encoding by setting the elements that were as 0 in the standard one-hot encoding as 0.1 to make contributions to the propagation.…”

“…In terms of aggregating multiple feature vectors, EI-Sappagh et al [24] used bidirectional long short-term memory (biLSTM) models to handle the vectorized time-series features, and Lu et al [18] applied attention-based MIL to accumulate the feature vectors in bags to bag-level representations. As for the image feature maps learned by CNNs, these feature maps could be used for fusion directly in order to keep the spatial information [59,68], vectorized with pooling layers [16,34,35], or split by pixel/voxels as the tokens to feed transformers [58].…”

“…Unimodal feature extraction can be either unsupervised or supervised. Note that if the unimodal features are trained prior to fusion, it is possible to train the unimodal model with the maximum number of available samples of each modality for better unimodal model performance and hopefully better unimodal features to benfit the multimodal performance [35,67,84]. Regarding the relationship of unimodal feature extraction with the fusion, the unimodal feature extraction section can be independent to the fusion section [35,87], which is known as early-level fusion, or can be trained from scratch or finetuned with the fusion section end-to-end [20,31] as the intermediate-level fusion.…”