Multi-Constraint Latent Representation Learning for Prognosis Analysis Using Multi-Modal Data

Ning, Zhenyuan; Lin, Zhiping; Xiao, Qing; Du, Denghui; Feng, Qianjin; Chen, Wufan

doi:10.1109/tnnls.2021.3112194

Cited by 12 publications

(8 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, by following previous work, we concatenate predicted risk values from the test sets in 5-fold cross-validation, which are then plotted against the respective survival time. Patients of each dataset are divided into low and high-risk groups based on the on the median of risk indices as the threshold ( Ning et al , 2021a ). We can find that CAMR successfully divides the LGG patients into low and high-risk groups with optimal patient stratification ( P = 2.2204e−16).…”

Section: Resultsmentioning

confidence: 99%

“…The reasons why we choose pre-extract features for a deep learning model can listed as follows. At first, limited by the computational resources, using CNN to extract features from histopathological images be expensive or even impractical ( Ning et al , 2021a ). Secondly, the features retrieved by CNN may result in overfitting in small cohorts ( Boehm et al , 2022 ), which may limit the further improvement of performance of cancer survival prediction.…”

Section: Discussionmentioning

confidence: 99%

“… Dong et al (2019) propose an efficient improved gcForest model named MLW-gcForest to solve the problem of adenocarcinoma staging by taking genetic data as inputs. Ning et al (2021a ) present a multi-constraint latent representation learning method called McLR to achieve promising performance of cancer prognosis by learning a common subspace. Besides, Ning et al (2021b ) introduce an impressive method called Relation-aware Shared Representation learning (RaSR) to unify both representation learning and prognosis modeling into a joint framework and improve the performance of cancer prognosis.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

CAMR: cross-aligned multimodal representation learning for cancer survival prediction

Shi

Wang

et al. 2023

Bioinformatics

View full text Add to dashboard Cite

Motivation Accurately predicting cancer survival is crucial for helping clinicians to plan appropriate treatments, which largely improves the life quality of cancer patients and spares the related medical costs. Recent advances of survival prediction methods suggest that integrating complementary information from different modalities, e.g., histopathological images and genomic data, plays a key role in enhancing predictive performance. Despite promising results obtained by existing multimodal methods, the disparate and heterogeneous characteristics of multimodal data cause so-called modality gap problem, which brings in dramatically diverse modality representations in feature space. Consequently, detrimental modality gaps make it difficult for comprehensive integration of multimodal information via representation learning, and therefore pose a great challenge to further improvements of cancer survival prediction. Results To solve the above problems, we propose a novel method called cross-aligned multimodal representation learning (CAMR), which generates both modality-invariant and -specific representations for more accurate cancer survival prediction. Specifically, a cross-modality representation alignment learning network is introduced to reduce modality gaps by effectively learning modality-invariant representations in a common subspace, which is achieved by aligning the distributions of different modality representations through adversarial training. Besides, we adopt a cross-modality fusion module to fuse modality-invariant representations into a unified cross-modality representation for each patient. Meanwhile, CAMR learns modality-specific representations which complement modality-invariant representations and therefore provides a holistic view of the multimodal data for cancer survival prediction. Comprehensive experiment results demonstrate that CAMR can successfully narrow modality gaps and consistently yields better performance than other survival prediction methods using multimodal data. Availability CAMR is freely available at https://github.com/wxq-ustc/CAMR Supplementary information Supplementary data are available at Bioinformatics online.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CAMR: cross-aligned multimodal representation learning for cancer survival prediction

Shi

Wang

et al. 2023

Bioinformatics

View full text Add to dashboard Cite

show abstract

“…Multi-modal data provides richer information than a single modality does, which has attracted more and more attention in both natural data processing and medical data processing fields, such as visual question answering (VQA) [33], RGB-D object recognition [34] and pathogenomics for prognosis analysis [35]. For most of the above tasks, there exists a huge gap between two different modalities, such as {text, image, speech} and {whole slide image, genomic data}.…”

Section: Multi-modal Fusion Modelsmentioning

confidence: 99%

CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer With Modality-Correlated Cross-Attention for Brain Tumor Segmentation

Lin

et al. 2023

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

Brain tumor segmentation (BTS) in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges as well as the advances of CNN and Transformer algorithms, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. However, existing studies hardly consider how to fuse the multimodality images in a reasonable manner. In this paper, we leverage the clinical knowledge of how radiologists diagnose brain tumors from multiple MRI modalities and propose a clinical knowledge-driven brain tumor segmentation model, called CKD-TransBTS. Instead of directly concatenating all the modalities, we re-organize the input modalities by separating them into two groups according to the imaging principle of MRI. A dual-branch hybrid encoder with the proposed modality-correlated crossattention block (MCCA) is designed to extract the multimodality image features. The proposed model inherits the strengths from both Transformer and CNN with the local feature representation ability for precise lesion boundaries and long-range feature extraction for 3D volumetric images. To bridge the gap between Transformer and CNN features, we propose a Trans&CNN Feature Calibration block (TCFC)

show abstract

“…Features with the mean absolute correlation higher than 0.9 were regarded as redundant, and the one with lower C-index was eliminated. Then, considering Cox-based models can benefit from regression constraint of the actual length of the observed time (Ning et al 2021), minimum redundancy maximum relevance (MRMR) for multivariable regression of overall survival time based on non-censored cases was used to identify the most time-related features, and the 30 top-ranking features were kept for sequential backward selection. Finally, considering there are 161 patients who experienced with death/event in the training dataset, and the number of features should not exceed 10% of the number of events (Peduzzi et al 1996), a subset of 15 features was determined by backward selection for subsequent model construction.…”

Section: Radiomics Feature and Clinical Factor Selectionmentioning

confidence: 99%

Optimal batch determination for improved harmonization and prognostication of multi-center PET/CT radiomics feature in head and neck cancer

Wu,

Liu,

Peng

et al. 2023

Phys. Med. Biol.

View full text Add to dashboard Cite

Objective To determine the optimal approach for identifying and mitigating batch effects in PET/CT radiomics features, and further improve the prognosis of patients with head and neck cancer (HNC), this study investigated the performance of three batch harmonization methods. Approach Unsupervised harmonization identified the batch labels by K-means clustering. Supervised harmonization regarding the image acquisition factors (center, manufacturer, scanner, filter kernel) as known/given batch labels, and Combat harmonization was then implemented separately and sequentially based on the batch labels, i.e. harmonizing features among batches determined by each factor individually or harmonizing features among batches determined by multiple factors successively. Extensive experiments were conducted to predict overall survival (OS) on public PET/CT datasets that contain 800 patients from 9 centers. Main results In the external validation cohort, results show that compared to original models without harmonization, Combat harmonization would be beneficial in OS prediction with C-index of 0.687-0.740 vs. 0.684-0.767. Supervised harmonization slightly outperformed unsupervised harmonization in all models (C-index: 0.692-0.767 vs. 0.684-0.750). Separate harmonization outperformed sequential harmonization in CT_m+clinic and CT_cm+clinic models with C-index of 0.752 and 0.722, respectively, while sequential harmonization involved clinical features in PET_rs+clinic model further improving the performance and achieving the highest C-index of 0.767. Significance Optimal batch determination especially sequential harmonization for Combat holds the potential to improve the prognostic power of radiomics model in multi-center HNC dataset with PET/CT imaging.

show abstract

Multi-Constraint Latent Representation Learning for Prognosis Analysis Using Multi-Modal Data

Cited by 12 publications

References 46 publications

CAMR: cross-aligned multimodal representation learning for cancer survival prediction

CAMR: cross-aligned multimodal representation learning for cancer survival prediction

CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer With Modality-Correlated Cross-Attention for Brain Tumor Segmentation

Optimal batch determination for improved harmonization and prognostication of multi-center PET/CT radiomics feature in head and neck cancer

Contact Info

Product

Resources

About