Integrated multimodal artificial intelligence framework for healthcare applications

Abstract

Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database (N = 34,537 samples) containing 7279 unique hospitalizations and 6485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6–33%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48 h mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.

High night-to-night variability in sleep apnea severity is associated with uncontrolled hypertension

“…Conceptually, these data could also be synchronized via electronic health records databases, and via the development of new digital health tools to automatically capture more diverse aspects of a person’s health. Multimodal inputs to predictive algorithms (i.e., using health data from different sources) have been recently shown to better predict health outcomes compared to single-source approaches across 12 predictive tasks, including 10 distinct chest pathology diagnoses, hospital length-of-stay, and 48 h mortality predictions 38 . Similarly, the field of sleep medicine may benefit from multi-input clinical data where daytime symptoms, overall clinical history, and multi-sensor recordings could be used to better predict health outcomes and treatment response.…”

Telehealth’s Role Enabling Sustainable Innovation and Circular Economies in Health

“…A circular economy aims to drive sustainability, equity, and digital inclusion, 27 translating to further transformation cycles and resilience. Accelerating the use of AI for knowledge discovery is needed [28][29][30][31][32][33][34][35] to encourage and foster cooperation and implement industry standards that translate knowledge discovery into readily available, high-quality service interventions.…”

Privacy-preserving machine learning for healthcare: open challenges and future perspectives

“…Current advances in ML for healthcare are moving towards multi-modal learning, where several sources of information are combined to improve performance (Ramachandram & Taylor, 2017;Soenksen et al, 2022). This approach not only tends to provide better performance but also ensures a comprehensive understanding of the different physiological variables involved in studying and modeling the development of human biology and pathology.…”