Co-Learning Feature Fusion Maps From PET-CT Images of Lung Cancer

Abstract

The analysis of multi-modality positron emission tomography and computed tomography (PET-CT) images requires combining the sensitivity of PET to detect abnormal regions with anatomical localization from CT. However, current methods for PET-CT image analysis either process the modalities separately or fuse information from each modality based on knowledge about the image analysis task. These methods generally do not consider the spatially varying visual characteristics that encode different information across the different modalities, which have different priorities at different locations. For example, a high abnormal PET uptake in the lungs is more meaningful for tumor detection than physiological PET uptake in the heart. Our aim is to improve fusion of the complementary information in multi-modality PET-CT with a new supervised convolutional neural network (CNN) that learns to fuse complementary information for multi-modality medical image analysis. Our CNN first encodes modality-specific features and then uses them to derive a spatially varying fusion map that quantifies the relative importance of each modality's features across different spatial locations. These fusion maps are then multiplied with the modality-specific feature maps to obtain a representation of the complementary multi-modality information at different locations, which can then be used for image analysis, e.g. region detection. We evaluated our CNN on a region detection problem using a dataset of PET-CT images of lung cancer. We compared our method to baseline techniques for multi-modality image analysis (pre-fused inputs, multi-branch techniques, multichannel techniques) and demonstrated that our approach had a significantly higher accuracy (p < 0.05) than the baselines.

FSS-2019-nCov: A deep learning architecture for semi-supervised few-shot segmentation of COVID-19 infection

“…For instance, Kumar et al. [30] introduced a new supervised CNN to fuse complementary multi-modality information from lung cancer scans. Ozdemir et al.…”

Multi-modality medical image fusion technique using multi-objective differential evolution based deep neural networks

“…Kumar et al. ( 2020 ) proposed a co-learning based fusion maps for obtained more efficient multi-modality fused biomedical images. A convolutional neural network (CNN) was also used for the prediction and segmentation of potential objects.…”

Application of Image Fusion in Diagnosis and Treatment of Liver Cancer

“…CNN can decompose the original images to high frequency and low frequency images [146], and select the rule of regional matching to fuse the two high frequency and low frequency images to get the final fusion images. Kumar et al [147] developed a supervised CNN to learn to merge the data from PET-CT images of lung cancer. CNN has also been applied to fuse medical images MRI/CT, MRI/SPECT, multiparametric MR images [148] and PET/MRI [149].…”

“…For the fusion of PET-MRI for clinical applications, the wavelet transformation-based method [66,73,74,130], IHS-PCA [31] and deep learning methods [151,154] are generally used. On the other hand, wavelet transformation-based methods [52,57,168,169] and deep learning [147] are generally applied for the fusion of PET-CT. Here, we show an example in Figure 5, which compares the accuracy of the fused image of PET/MRI and single modal MRI in the correct identification of a patient with liver lesions [183].…”