Multimodal machine learning in precision health: A scoping review

Abstract

Machine learning is frequently being leveraged to tackle problems in the health sector including utilization for clinical decision-support. Its use has historically been focused on single modal data. Attempts to improve prediction and mimic the multimodal nature of clinical expert decision-making has been met in the biomedical field of machine learning by fusing disparate data. This review was conducted to summarize the current studies in this field and identify topics ripe for future research. We conducted this review in accordance with the PRISMA extension for Scoping Reviews to characterize multi-modal data fusion in health. Search strings were established and used in databases: PubMed, Google Scholar, and IEEEXplore from 2011 to 2021. A final set of 128 articles were included in the analysis. The most common health areas utilizing multi-modal methods were neurology and oncology. Early fusion was the most common data merging strategy. Notably, there was an improvement in predictive performance when using data fusion. Lacking from the papers were clear clinical deployment strategies, FDA-approval, and analysis of how using multimodal approaches from diverse sub-populations may improve biases and healthcare disparities. These findings provide a summary on multimodal data fusion as applied to health diagnosis/prognosis problems. Few papers compared the outputs of a multimodal approach with a unimodal prediction. However, those that did achieved an average increase of 6.4% in predictive accuracy. Multi-modal machine learning, while more robust in its estimations over unimodal methods, has drawbacks in its scalability and the time-consuming nature of information concatenation.

Wearable-based Physical Activity as a Digital Biomarker of Inflammation, Biological Age, and Mortality: A Hierarchical Clustering Analysis of NHANES 2011-2014

“…Additionally, it would allow healthcare professionals to gain a deeper understanding of patients and atrisk populations, considering a variety of factors that impact their health status, disease risks, and medical conditions. Furthermore, the application of machine learning (ML) to digital biomarker data elucidates hidden patterns as digital phenotypes and facilitates subpopulation identi cation 17 . Conventional, expert-driven classi cation of disease or at-risk populations is limited by a lack of agreed ways of knowing the number of natural clusters in the populations of interest and determining the variables on which to base segmentation 18,19 .…”

A Scalable Framework for Closed-Loop Neuromodulation with Deep Learning

“…dearth of large open-sourced neuromodulation datasets, however with advances in data modeling and augmentation 59 techniques (diffusion models, variational autoencoders -VAEs, GANs etc. [60][61][62][63][64] ), transfer learning 65,66 , and improved data integration and sharing infrastructure with data harmonization 67 , these deficits will be addressed.…”

A guide to multi-omics data collection and integration for translational medicine

“…These layers could capture the correlations and complex biological relationships within the datasets [39] . This representation is envisioned to be able to combine heterogeneous datasets more efficiently [40] , [41] . SubtypeGAN [42] is based on a generative adversarial network consisting of multi-input and multi-output layers and a shared layer representing the multi-omics input.…”