Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database (N = 34,537 samples) containing 7279 unique hospitalizations and 6485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6–33%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48 h mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.

Drought is the most expensive natural hazard and one of the deadliest. While drought propagation through standardised indices has been extensively studied at the regional scale, global scale drought propagation, and particularly quantifying the space and time variability, is still a challenging task. Quantifying the space time variability is crucial to understand how droughts have changed globally in order to cope with their impacts. In particular, better understanding of the propagation of drought through the climate, vegetation and hydrological subsystems can improve decision making and preparedness. This study maps spatial temporal drought propagation through different subsystems at the global scale over the last decades. The standardised precipitation index (SPI) based on the gamma distribution, the standardised precipitation evapotranspiration index (SPEI) based on the log-logistic distribution, the standardised vegetation index (SVI) based on z-scores, and the standardised runoff index (SRI) based on empirical runoff probabilities were quantified. Additionally, drought characteristics, including duration, severity and intensity were estimated. Propagation combined the delay in response in the subsystems using drought characteristics, and trends in time were analysed. All these were calculated at 0.05 to 0.25 arc degree pixels. In general, drought propagates rapidly to the response in runoff and streamflow, and a with longer delay in the vegetation. However, this response varies spatially across the globe and depending on the observation scale, and amplifies progressively in duration and severity across large regions from the meteorological to the agricultural/ecological and hydrologic subsystems, while attenuating in intensity. Significant differences exist between major Köppen climate groups in drought characteristics and propagation. Patterns show intensification of drought severity and propagation affecting vegetation and hydrology in regions of southern South America, Australia, and South West Africa.

No abstract