Fusing of Medical Images and Reports in Diagnostics of Brain Diseases

Vatian, Aleksandra; Gusarova, Natalia; Dobrenko, Natalia; Klochkov, Anton; Nigmatullin, Niyaz; Lobantsev, Artem; Shalyto, Anatoly

doi:10.1145/3357777.3357793

Cited by 6 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, it is worth mentioning that among the studies that produced 100% results, 5 of the studies used MRI, while the other studies used different datatypes like OCT, ERG, and clinical features. Vatian et al [ 104 ] used MRI and the radiologist notes to train the model. That study combined text mining with image analysis.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Multiple Sclerosis Diagnosis Using Machine Learning and Deep Learning: Challenges and Opportunities

Aslam

Khan

Bashamakh

et al. 2022

Sensors

View full text Add to dashboard Cite

Multiple Sclerosis (MS) is a disease that impacts the central nervous system (CNS), which can lead to brain, spinal cord, and optic nerve problems. A total of 2.8 million are estimated to suffer from MS. Globally, a new case of MS is reported every five minutes. In this review, we discuss the proposed approaches to diagnosing MS using machine learning (ML) published between 2011 and 2022. Numerous models have been developed using different types of data, including magnetic resonance imaging (MRI) and clinical data. We identified the methods that achieved the best results in diagnosing MS. The most implemented approaches are SVM, RF, and CNN. Moreover, we discussed the challenges and opportunities in MS diagnosis to improve AI systems to enable researchers and practitioners to enhance their approaches and improve the automated diagnosis of MS. The challenges faced by automated MS diagnosis include difficulty distinguishing the disease from other diseases showing similar symptoms, protecting the confidentiality of the patients’ data, achieving reliable ML models that are also easily understood by non-experts, and the difficulty of collecting a large reliable dataset. Moreover, we discussed several opportunities in the field such as the implementation of secure platforms, employing better AI solutions, developing better disease prognosis systems, combining more than one data type for better MS prediction and using OCT data for diagnosis, utilizing larger, multi-center datasets to improve the reliability of the developed models, and commercialization.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Vatian et al [ 104 ] used a combination of MRI scans and clinical data to diagnose MS. They focused on fusing information acquired from a collection of MRI scans and clinical data from medical reports corresponding to these images collected from 19 patients.…”

Section: Related Studiesmentioning

confidence: 99%

Multiple Sclerosis Diagnosis Using Machine Learning and Deep Learning: Challenges and Opportunities

Aslam

Khan

Bashamakh

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…In conclusion of the review, it should be said that there are few works devoted directly to the fusion of texts and images for the diagnostics of particular diseases [18,48] -this emphasizes the urgency of the problem.…”

Section: Background and Related Workmentioning

confidence: 99%

Untitled

2020

ICS

View full text Add to dashboard Cite

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Comparative assessment of text-image fusion models for medical diagnostics

Gusarova

Lobantsev

Vatian

et al. 2020

ICS

Self Cite

View full text Add to dashboard Cite

Introduction: Information overload and complexity are characteristic of decision-making in medicine. In these conditions information fusion techniques are effective. For the diagnosis and treatment of pneumonia using x-ray images and accompaniyng free-text radiologists reports, it is promising to use text-image fusion. Purpose: Development of a method for fusing text with an image in the treatment and diagnosis of pneumonia using neural networks. Methods: We used MIMIC-CXR dataset, the SEResNeXt101-32x4d for images feature extracting and the Bio-ClinicalBERT model followed by ContextLSTM layer for text featureextracting. We compared five architectures in the conducted experiment: image classifier, report classifier and three scenarios of the fusion, namely late fusion, middle fusion and early fusion. Results: We got an absolute excess of metrics (ROC AUC = 0.9933, PR AUC = 0.9907) when using an early fusion classifier (ROC AUC = 0.9921, PR AUC = 0.9889) even over the idealized case of text classifier (that is, without taking into account possible errors of the radiologist). The network training time ranged from 20 minutes for late fusion to 9 hours and 45 minutes for early fusion. Based on Class Activation Map technique we graphically showed that the image feature extractor in the fused classification scenario still learns discriminative regions for pneumonia classification problem. Discussion: Fusing text and images increases the likelihood of correct image classification compared to only image classification. The proposed combined image-report classifier trained with the early-fusion method gives better performance than individual classifiers in the pneumonia classification problem. However, it is worth considering that better results cost the training time and required computation resources. Report-based training is much faster in training and less demanding for computation capacity.

show abstract

Fusing of Medical Images and Reports in Diagnostics of Brain Diseases

Cited by 6 publications

References 18 publications

Multiple Sclerosis Diagnosis Using Machine Learning and Deep Learning: Challenges and Opportunities

Multiple Sclerosis Diagnosis Using Machine Learning and Deep Learning: Challenges and Opportunities

Untitled

Comparative assessment of text-image fusion models for medical diagnostics

Contact Info

Product

Resources

About