Richer fusion network for breast cancer classification based on multimodal data

Abstract

Background Deep learning algorithms significantly improve the accuracy of pathological image classification, but the accuracy of breast cancer classification using only single-mode pathological images still cannot meet the needs of clinical practice. Inspired by the real scenario of pathologists reading pathological images for diagnosis, we integrate pathological images and structured data extracted from clinical electronic medical record (EMR) to further improve the accuracy of breast cancer classification. Methods In this paper, we propose a new richer fusion network for the classification of benign and malignant breast cancer based on multimodal data. To make pathological image can be integrated more sufficient with structured EMR data, we proposed a method to extract richer multilevel feature representation of the pathological image from multiple convolutional layers. Meanwhile, to minimize the information loss for each modality before data fusion, we use the denoising autoencoder as a way to increase the low-dimensional structured EMR data to high-dimensional, instead of reducing the high-dimensional image data to low-dimensional before data fusion. In addition, denoising autoencoder naturally generalizes our method to make the accurate prediction with partially missing structured EMR data. Results The experimental results show that the proposed method is superior to the most advanced method in terms of the average classification accuracy (92.9%). In addition, we have released a dataset containing structured data from 185 patients that were extracted from EMR and 3764 paired pathological images of breast cancer, which can be publicly downloaded from http://ear.ict.ac.cn/?page_id=1663. Conclusions We utilized a new richer fusion network to integrate highly heterogeneous data to leverage the structured EMR data to improve the accuracy of pathological image classification. Therefore, the application of automatic breast cancer classification algorithms in clinical practice becomes possible. Due to the generality of the proposed fusion method, it can be straightforwardly extended to the fusion of other structured data and unstructured data.

Artificial intelligence-based methods for fusion of electronic health records and imaging data

“…To accomplish this, various methods were used in the studies, including neural network-based features extraction, data generation through software, or manual extraction of features. Out of the 22 early fusion studies, 19 studies 12 , 13 , 15 , 25 , 33 – 36 , 39 , 41 – 45 , 50 – 53 used manual or software-based imaging features, and 3 studies used neural network-based architectures to extract imaging features before combining with other clinical data modality 16 , 18 , 54 . Six out of the 19 studies that used manual or software-based features reduced the feature dimension before concatenating the two modalities’ features using different methods 25 , 36 , 45 , 50 – 52 .…”

“…Fourteen early fusion studies evaluated their fusion models’ performance against that of single modality models 12 , 13 , 15 , 16 , 18 , 25 , 32 – 34 , 36 , 41 – 44 , 51 . As a result, 13 of these studies exhibited a better performance for fusion when compared with their imaging-only and clinical-only counterparts 12 , 13 , 15 , 16 , 18 , 25 , 32 – 34 , 41 – 44 , 51 .…”

“…The most common applied clinical outcome in the included studies was the diagnosis, reported in 20 ( ) studies. In these studies, EHRs were combined with medical imaging to diagnose a spectrum of diseases including neurological disorders ( ) 4 , 13 – 15 , 32 , 37 , 42 , 49 , 50 , psychiatric disorders ( ) 33 , 36 , CVD ( ) 54 – 56 , Cancer ( ) 16 , 55 , and four studies for other different diseases 18 , 19 , 40 , 53 . Specifically, most of the studies that focused on detecting neurological diseases were for AD ( ) 13 – 15 , 49 , and MCI ( ) 37 , 42 , 50 , 51 .…”

“… Fusion implementation : Early fusion was the most commonly used technique in most applications for multimodal learning. Before fusing 1D EHRs data with image data in 2D or 3D, images data was converted to a 1D vector by extracting high-level representations using manual or software-generated features 12 , 13 , 15 , 25 , 33 – 36 , 39 , 41 – 45 , 50 – 53 , or CNN-extracted features 8 , 16 , 54 . The learned imaging features from CNN often resulted in better task-specific performance than manually or software-derived features 64 .…”

“…This can be seen in Alzheimer’s disease diagnosis and prediction 12 – 15 when imaging data were combined with specific lab test results and demographic data as inputs to ML models, and better performance was achieved than the single-source models. Similarly, fusing pathological images with patient demographic data observed an increase in performance in comparison with single modality models for breast cancer diagnosis 16 . Several studies found similar advantages in various medical imaging applications, including diabetic retinopathy prediction, COVID-19 detection, and glaucoma diagnosis 17 – 19 .…”