Developing a Predictive Model for Asthma-Related Hospital Encounters in Patients With Asthma in a Large, Integrated Health Care System: Secondary Analysis

Abstract

Background Asthma causes numerous hospital encounters annually, including emergency department visits and hospitalizations. To improve patient outcomes and reduce the number of these encounters, predictive models are widely used to prospectively pinpoint high-risk patients with asthma for preventive care via care management. However, previous models do not have adequate accuracy to achieve this goal well. Adopting the modeling guideline for checking extensive candidate features, we recently constructed a machine learning model on Intermountain Healthcare data to predict asthma-related hospital encounters in patients with asthma. Although this model is more accurate than the previous models, whether our modeling guideline is generalizable to other health care systems remains unknown. Objective This study aims to assess the generalizability of our modeling guideline to Kaiser Permanente Southern California (KPSC). Methods The patient cohort included a random sample of 70.00% (397,858/568,369) of patients with asthma who were enrolled in a KPSC health plan for any duration between 2015 and 2018. We produced a machine learning model via a secondary analysis of 987,506 KPSC data instances from 2012 to 2017 and by checking 337 candidate features to project asthma-related hospital encounters in the following 12-month period in patients with asthma. Results Our model reached an area under the receiver operating characteristic curve of 0.820. When the cutoff point for binary classification was placed at the top 10.00% (20,474/204,744) of patients with asthma having the largest predicted risk, our model achieved an accuracy of 90.08% (184,435/204,744), a sensitivity of 51.90% (2259/4353), and a specificity of 90.91% (182,176/200,391). Conclusions Our modeling guideline exhibited acceptable generalizability to KPSC and resulted in a model that is more accurate than those formerly built by others. After further enhancement, our model could be used to guide asthma care management. International Registered Report Identifier (IRRID) RR2-10.2196/resprot.5039

Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis

“…We adopted the same patient cohort from our previous KPSC predictive model paper [25]: a random sample of 70% of patients with asthma who had a KPSC health plan for any period between 2015 and 2018. This sample size is the largest one permitted by KPSC for sharing its data with another non-Kaiser Permanente institution for research.…”

“…We adopted the same prediction target as our prior KPSC predictive model paper [25]. For every patient deemed to have asthma in a specific year, the indicator of any asthma-related hospital visit in the succeeding year is the outcome.…”

“…We adopted the same administrative and clinical data set from our prior KPSC predictive model paper [25]. Obtained from KPSC's research data warehouse, this structured data set covered our patient cohort's visits at KPSC between 2010 and 2018.…”

“…Our KPSC model [25] uses the XGBoost classification algorithm [23] and 221 features to forecast asthma-related hospital visits in the succeeding year in patients with asthma. These features are listed in our previous KPSC predictive model paper [25], were computed on the structured attributes in our data set, and cover various characteristics such as patient demographics, medications, visits, diagnoses, vital signs, procedures, and laboratory tests. An example feature is the total number of asthma relievers that the patient filled in the previous 12 months.…”

“…In addition, patient outcomes deteriorate, whereas health care costs increase. To address this problem, we recently considered many candidate features and developed two extreme gradient boosting (XGBoost) [23] machine learning models, one on Intermountain Healthcare data [24] and the other on Kaiser Permanente Southern California (KPSC) data [25], to forecast asthma-related hospital visits in the succeeding 12 months among patients with asthma with a higher accuracy. As is typical for machine learning approaches, these two models do not explain their forecasting results.…”