A new paradigm for accelerating clinical data science at Stanford Medicine

Datta, Somalee; Posada, Jose; Olson, Garrick; Li, Wencheng; O'Reilly, Conor; Balraj, Deepa; Mesterhazy, Joseph; Pallas, Joseph; Desai, Priyamvada; Shah, Nigam H.

doi:10.48550/arxiv.2003.10534

Cited by 11 publications

(16 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used data from the STAnford medicine Research data Repository (STARR)[17]. Data in STARR are routinely collected in the EHR of Stanford Medical Center, comprised of Stanford Health Care (primarily adult-directed care) and Lucile Packard Children’s Hospital (primarily pediatric-directed care).…”

Section: Methodsmentioning

confidence: 99%

EHR Foundation Models Improve Robustness in the Presence of Temporal Distribution Shift

Guo

Steinberg

Fleming

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

BackgroundTemporal distribution shift negatively impacts the performance of clinical prediction models over time. Pretraining foundation models using self-supervised learning on electronic health records (EHR) may be effective in acquiring informative global patterns that can improve the robustness of task-specific models.ObjectiveTo evaluate the utility of EHR foundation models in improving the in-distribution (ID) and out-of-distribution (OOD) performance of clinical prediction models.MethodsThe cohort consisted of adult inpatients admitted between 2009-2021. Gated recurrent unit (GRU)- and transformer (TRANS)-based foundation models were pretrained on EHR of patients admitted between 2009-2012 and were subsequently used to construct patient representations (CLMBR). These representations were used to learn logistic regression models (CLMBRGRU and CLMBRTRANS) to predict hospital mortality, long length of stay, 30-day readmission, and ICU admission. We compared CLMBRGRU and CLMBRTRANS with baseline logistic regression models learned on count-based representations (count-LR) and end-to-end (ETE) GRU and transformer models in ID (2009-2012) and OOD (2013-2021) year groups. Performance was measured using area-under-the-receiver-operating-characteristic curve, area- under-the-precision-recall curve, and absolute calibration error.ResultsModels trained on CLMBR generally showed better discrimination relative to count-LR in both ID and OOD year groups. In addition, they often matched or were better than their ETE counterparts. Finally, foundation models’ performance in the self-supervised learning task tracked closely with the ID and OOD performance of the downstream models.ConclusionsThese results suggest that pretraining foundation models on electronic health records is a useful approach for developing clinical prediction models that perform well in the presence of temporal distribution shift.

show abstract

Section: Methodsmentioning

confidence: 99%

EHR Foundation Models Improve Robustness in the Presence of Temporal Distribution Shift

Guo

Steinberg

Fleming

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The STARR is a clinical warehouse that contains approximately 150 million encounter records from the Stanford Health Care and Lucile Packard Children's Hospital. This database comprises longitudinal data on patient and provider information, diagnoses, procedures, medication orders, and lab results that have been normalized to the Observational Medical Outcomes Partnership (OMOP) CDM version 5.3.1 [31], sourced from inpatient and outpatient clinical encounters that occurred between 1990 and 2021. The use of this data was conducted in accordance with all relevant guidelines and regulations.…”

Section: Data Sources and Formulation Of The Study Cohortmentioning

confidence: 99%

Clinical Utility Gains from Incorporating Comorbidity and Geographic Location Information into Risk Estimation Equations for Atherosclerotic Cardiovascular Disease

Xu¹,

Foryciarz²,

Steinberg³

et al. 2022

Preprint

View full text Add to dashboard Cite

Objective: There are several efforts to re-learn the 2013 ACC/AHA pooled cohort equations (PCE) for patients with specific comorbidities and geographic locations. With over 363 customized risk models in the literature, we aim to evaluate such revised models to determine if the performance improvements translate to gains in clinical utility.Methods: We re-train a baseline PCE using the ACC/AHA PCE variables and revise it to incorporate subject-level geographic location and comorbidity information. We apply fixed effects, random effects, and extreme gradient boosting models to handle the correlation and heterogeneity induced by locations. Models are trained using 2,464,522 claims records from Optum © 's Clinformatics ® Data Mart and validated in the hold-out set (N=1,056,224). We evaluate models' performance overall and across subgroups defined by the presence or absence of chronic kidney disease (CKD) or rheumatoid arthritis (RA) and geographic locations. We evaluate models' expected net benefit using decision curve analysis and models' statistical properties using several discrimination and calibration metrics.Results: The baseline PCE is miscalibrated overall, in patients with CKD or RA, and locations with small populations. Our revised models improved both the overall (GND P-value=0.41) and subgroup calibration but only enhanced net benefit in the underrepresented subgroups. The gains are larger in the subgroups with comorbidities and heterogeneous across geographic locations.Conclusions: Revising the PCE with comorbidity and location information significantly enhanced models' calibration; however, such improvements do not necessarily translate to clinical gains. Thus, we recommend future works to quantify the consequences from using risk calculators to guide clinical decisions.

show abstract

“…This database contains structured longitudinal data in the the form of diagnoses, procedures, medications, and laboratory tests that have been mapped to standard concept identifiers in the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) version 5.3.1 [80][81][82]. De-identified clinical notes with clinical concepts extracted and annotated with negation and family-history detection are also made available [79].…”

Section: Datasetsmentioning

confidence: 99%

“…For lab test results, we construct additional features for lab results above and below the corresponding reference range, if available, and indicators for whether the result belongs to the empirical quintiles for the corresponding lab result observed in the time interval. For STARR data, we include additional features for the presence of clinical concepts derived from clinical notes, modified by an indicator for whether the concept corresponds to a present and positive mention [79]. For MIMIC-III, we do not use diagnoses or clinical notes to derive features.…”

Section: Feature Extractionmentioning

confidence: 99%

“…STARR The Stanford Medicine Research Data Repository (STARR) [79] is a clinical data warehouse containing records from approximately three million patients from Stanford Hospitals and Clinics and the Lucile Packard Children's Hospital for inpatient and outpatient clinical encounters that occurred between 1990 and 2020. This database contains structured longitudinal data in the the form of diagnoses, procedures, medications, and laboratory tests that have been mapped to standard concept identifiers in the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) version 5.3.1 [80][81][82].…”

Section: Datasetsmentioning

confidence: 99%

See 1 more Smart Citation

An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction

Pfohl,

Foryciarz,

Shah

2020

Preprint

View full text Add to dashboard Cite

The use of machine learning to guide clinical decision making has the potential to worsen existing health disparities. Several recent works frame the problem as that of algorithmic fairness, a framework that has attracted considerable attention and criticism. However, the appropriateness of this framework is unclear due to both ethical as well as technical considerations, the latter of which include trade-offs between measures of fairness and model performance that are not well-understood for predictive models of clinical outcomes. To inform the ongoing debate, we conduct an empirical study to characterize the impact of penalizing group fairness violations on an array of measures of model performance and group fairness. We repeat the analyses across multiple observational healthcare databases, clinical outcomes, and sensitive attributes. We find that procedures that penalize differences between the distributions of predictions across groups induce nearly-universal degradation of multiple performance metrics within groups. On examining the secondary impact of these procedures, we observe heterogeneity of the effect of these procedures on measures of fairness in calibration and ranking across experimental conditions. Beyond the reported trade-offs, we emphasize that analyses of algorithmic fairness in healthcare lack the contextual grounding and causal awareness necessary to reason about the mechanisms that lead to health disparities, as well as about the potential of algorithmic fairness methods to counteract those mechanisms. In light of these limitations, we encourage researchers building predictive models for clinical use to step outside the algorithmic fairness frame and engage critically with the broader sociotechnical context surrounding the use of machine learning in healthcare.

show abstract

A new paradigm for accelerating clinical data science at Stanford Medicine

Cited by 11 publications

References 0 publications

EHR Foundation Models Improve Robustness in the Presence of Temporal Distribution Shift

EHR Foundation Models Improve Robustness in the Presence of Temporal Distribution Shift

Clinical Utility Gains from Incorporating Comorbidity and Geographic Location Information into Risk Estimation Equations for Atherosclerotic Cardiovascular Disease

An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction

Contact Info

Product

Resources

About