Reengineering a machine learning phenotype to adapt to the changing COVID-19 landscape: A study from the N3C and RECOVER consortia

Abstract

BackgroundIn 2021, we used the National COVID Cohort Collaborative (N3C) as part of the NIH RECOVER Initiative to develop a machine learning (ML) pipeline to identify patients with a high probability of having post-acute sequelae of SARS-CoV-2 infection (PASC), or Long COVID. However, the increased home testing, missing documentation, and reinfections that characterize the latter years of the pandemic necessitate reengineering our original model to account for these changes in the COVID-19 research landscape.MethodsOur updated XGBoost model gathers data for each patient in overlapping 100-day periods that progress through time, and issues a probability of Long COVID for each 100-day period. If a patient has known acute COVID-19 during any 100-day window (including reinfections), we censor the data from 7 days prior to the diagnosis/positive test date through 28 days after. These fixed time windows replace the prior model’s reliance on a documented COVID-19 index date to anchor its data collection, and are able to account for reinfections.ResultsThe updated model achieves an area under the receiver operating characteristic curve of 0.90. Precision and recall can be adjusted according to a given use case, depending on whether greater sensitivity or specificity is warranted.DiscussionBy eschewing the COVID-19 index date as an anchor point for analysis, we are now able to assess the probability of Long COVID among patients who may have tested at home, or with suspected (but untested) cases of COVID-19, or multiple SARS-CoV-2 reinfections. We view this exercise as a model for maintaining and updating any ML pipeline used for clinical research and operations.

Effect of Paxlovid Treatment During Acute COVID-19 on Long COVID Onset: An EHR-Based Target Trial Emulation from the N3C and RECOVER Consortia

“…The PASC computable phenotype may also misclassify patients. 29 For this reason, the confidence intervals around computable phenotype-based incidence estimates are likely too narrow. Vaccination status is poorly documented in most EHRs, which precluded its use as a covariate.…”

“…The model considers many more diagnosis codes than those included in the symptom clusters (see the “SNOMED Roll Up” section in the supplement of Crosskey et al, 2023), and a positive prediction may be based on other diagnosis codes. 11 Also, the computable phenotype model does not include a novelty restriction. For example, if a patient had a dyspnea diagnosis in the three years prior to index, a post-acute dyspnea diagnosis would not count for the respiratory symptom cluster, but it would be considered by the computable phenotype model.…”

“…The PASC computable phenotype may also misclassify patients. 47 For this reason, the confidence intervals around computable phenotype-based incidence estimates are likely too narrow.…”

“…We measured overall PASC incidence using a machine learning-based computable phenotype model, which gathers data for each patient in overlapping 100-day periods that progress through time, and issues a probability of PASC for each 100-day period. 29 The model was trained to classify whether patients have a U09.9 ("Post COVID-19 Condition") ICD-10 diagnosis code in each period, based on the patients' diagnoses during each period. Diagnoses surrounding known COVID-19 infections are blacked out.…”

Long COVID incidence in adults and children between 2020 and 2023: a real-world data study from the RECOVER Initiative

“…N3C’s definition identified a) earliest documentation of a U09.9 / B94.8 diagnosis code or b) patients with predicted PASC by a machine-learning based algorithm trained on patients with a U09.9 diagnosis. 25 PCORnet and PEDSnet applied rules-based definitions based on a combination of clinical input and data-driven analysis of new-onset diagnoses more common in patients with COVID-19 than without. 26 – 28 Generally, PCORnet classified patients based on the presence of a U09.9 or B94.8 code, or at least one incident PASC diagnosis.…”