Potential pitfalls in the use of real-world data for studying long COVID

Honerlaw, Jacqueline; Maripuri, Monika; Beaulieu‐Jones, Brendin R.; Baig, Huma S; L'Yi, Sehi; Ho, Yuk‐Lam; Morris, Michele; Panickan, Vidul Ayakulangara; Wang, Xuan; Weber, Griffin M.; Liao, Katherine P.; Visweswaran, Shyam; Tan, Bryce W.Q.; Yuan, Wei; Gehlenborg, Nils; Muralidhar, Sumitra; Aaron, James R.; Agapito, Giuseppe; Albayrak, Adem; Albi, Giuseppe; Alessiani, M.; Alloni, Anna; Amendola, Danilo F.; Angoulvant, François; Anthony, Li L. L. J.; Aronow, Bruce J.; Ashraf, Fatima; Atz, Andrew M.; Avillach, Paul; Azevedo, Paula S.; Balshi, James; Beaulieu‐Jones, Brett K.; Bell, Douglas S.; Bellasi, Antonio; Bellazzi, Riccardo; Benoît, Vincent; Beraghi, Michele; Bernal-Sobrino, José Luis; Bernaux, Mélodie; Bey, Romain; Bhatnagar, Surbhi; Blanco-Martínez, Alvar; Bonzel, Clara-Lea; Booth, John; Bòsari, Silvano; Bourgeois, Florence T.; Bradford, Robert L.; Bréant, Stéphane; Brown, Nicholas; Bruno, Raffaele; Bryant, William A; Bucalo, Mauro; Bucholz, Emily M.; Burgun, Anita; Cannataro, Mario; Carmona, Aldo; Caucheteux, Charlotte; Champ, Julien; Chen, Jin; Chen, Krista Y.; Chiovato, Luca; Chiudinelli, Lorenzo; Cimino, James J.; Colicchio, Tiago K.; Cormont, Sylvie; Cossin, Sébastien; Craig, Jean B.; Cruz-Bermúdez, Juan Luis; Cruz-Rojo, Jaime; Dagliati, Arianna; Daniar, Mohamad; Daniel, Christel; Das, Priyam; Devkota, Batsal; Dionne, Audrey; Duan, Rui; Dubiel, Julien; DuVall, Scott L.; Estève, Loïc; Estiri, Hossein; Fan, Shirley; Follett, Robert W; Ganslandt, Thomas; Barrio, Noelia García; Garmire, Lana X.; Getzen, Emily J.; Geva, Alon; Gradinger, Tobias; Gramfort, Alexandre; Griffier, Romain; Griffon, Nicolas; Grisel, Olivier; Gutiérrez-Sacristán, Alba; Han, Larry; Hanauer, David A.; Haverkamp, Christian; Hazard, Derek; He, Bing; Henderson, Darren W.; Hilka, Martin; Holmes, John H.; Hong, Chuan; Huling, Kenneth M.; Hutch, Meghan R.; Issitt, Richard; Jannot, Anne Sophie; Jouhet, Vianney; Kavuluru, Ramakanth; Keller, Mark S; Kennedy, Chris; Key, Daniel; Kirchoff, Katie; Klann, Jeffrey G.; Krantz, Ian D.; Kraska, Detlef; Krishnamurthy, Ashok; Le, Trang T.; Leblanc, Judith; Lemaître, G; Lenert, Leslie; Leprovost, Damien; Liu, Molei; Loh, Ne Hooi Will; Long, Qi; Lozano‐Zahonero, Sara; Luo, Yuan; Lynch, Kristine E.; Mahmood, Sadiqa; Maidlow, Sarah E.; Makoudjou, Adeline; Malovini, Alberto; Mandl, Kenneth D.; Mao, Chengsheng; Maram, Anupama; Martel, Patricia; Martins, Marcelo R.; Marwaha, Jayson S.; Masino, Aaron J.; Mazzitelli, Maria; Mensch, Arthur; Milano, Marianna; Minicucci, Marcos F.; Moal, Bertrand; Ahooyi, Taha Mohseni; Moore, Jason H.; Moraleda, Cinta; Morris, Jeffrey S.; Moshal, Karyn; Mousavi, Sajad; Mowery, Danielle L.; Murad, Douglas A.; Murphy, Shawn N.; Naughton, Thomas P.; Neto, Carlos Tadeu Breda; Neuraz, Antoine; Newburger, Jane W.; Ngiam, Kee Yuan; Njoroge, Wanjikũ; Norman, James B.; Obeid, Jihad S.; Okoshi, Marina Politi; Olson, Karen L.; Omenn, Gilbert S.; Orlova, Nina; Ostasiewski, Brian; Palmer, Nathan; Paris, Nicolas; Patel, Lav P.; Pedrera-Jiménez, Miguel; Pfaff, Emily R.; Pfaff, Ashley C.; Pillion, Danielle; Pizzimenti, Sara; Prokosch, Hans U.; Prudente, Robson; Prunotto, Andrea; Quirós‐González, Víctor; Ramoni, Rachel; Raskin, Maryna; Rieg, Siegbert; Roig-Domínguez, Gustavo; Rojo, Pablo; Rubio-Mayo, Paula; Sacchi, Paolo; Sáez, Carlos; Salamanca, Elisa; Samayamuthu, Malarkodi Jebathilagam; Sanchez-Pinto, L. Nelson; Sandrin, Arnaud; Santhanam, Nandhini; Santos, Janaina C. C.; Vidorreta, Fernando J Sanz; Savino, Maria; Schriver, Emily; Schubert, Petra; Schuettler, Juergen; Scudeller, Luigia; Sebire, Neil J; Serrano-Balazote, Pablo; Serre, Patricia; Serret-Larmande, Arnaud; Shah, Mohsin; Abad, Zahra Shakeri Hossein; Silvio, Domenick; Sliz, Piotr; Son, Jiyeon; Sonday, Charles; South, Andrew M.; Spiridou, Anastasia; Strasser, Zachary H.; Tan, Amelia L. M.; Tan, Byorn W.L.; Tanni, Suzana Erico; Taylor, Deanne; Terriza-Torres, Ana I.; Tibollo, Valentina; Tippmann, Patric; Toh, Emma M. S.; Torti, Carlo; Trecarichi, Enrico Maria; Tseng, Yi-Ju; Vallejos, Andrew K.; Varoquaux, Gaël; Vella, Margaret E.; Verdy, Guillaume; Vie, Jill-Jênn; Vitacca, Michele; Wagholikar, Kavishwar B.; Waitman, Lemuel R.; Wassermann, Demián; Wolkewitz, Martin; Wong, Scott W.; Xiong, Xin; Ye, Ye; Yehya, Nadir; Zambelli, Alberto; Zhang, Harrison G.; Zöller, Daniela; Zuccaro, Valentina; Zucco, Chiara; Kohane, Isaac S.; Xia, Zongqi; Cho, Kelly; Cai, Tianxi; Brat, Gabriel

doi:10.1038/s41591-023-02274-y

Cited by 17 publications

(11 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… 31 Thus, a sufficient follow-up length defining people ‘at risk for long COVID’ is extremely important to minimise misclassification or misdiagnosis of PCC. 32 …”

Section: Discussionmentioning

confidence: 99%

Definition and measurement of post-COVID-19 conditions in real-world practice: a global systematic literature review

Yang,

Markus,

Andersen

et al. 2024

BMJ Open

View full text Add to dashboard Cite

Post-COVID-19 conditions (PCC) is an umbrella term that encompasses a range of signs, symptoms and conditions present weeks after the acute phase of a SARS-CoV-2 infection. This systematic literature review summarises the heterogeneous methodology used to measure PCC across real-world studies and highlights trends by region, age group, PCC follow-up period and data source.MethodsMedline, EMBASE and the Cochrane Library were searched and supplemented with conference and grey literature searches. Eligible studies included individuals with (1) PCC or (2) a positive SARS-CoV-2 test or COVID-19 diagnosis who were followed over time. Included studies were published in English between 1 January 2020 and 14 November 2022.FindingsOf 291 publications included, 175 (60%) followed individuals with confirmed COVID-19 over time for PCC and 116 (40%) used a prespecified PCC definition. There was substantial heterogeneity in study design, geography, age group, PCC conditions/symptoms assessed and their classification and duration of follow-up. Among studies using a prespecified PCC definition, author-defined criteria (51%) were more common than criteria recommended by major public health organisations (19%). Measurement periods for PCC outcomes from date of acute COVID-19 test were primarily 3 to <6 months (39.2%), followed by 6 to <12 months (27.5%) and <3 months (22.9%). When classified by organ/system, constitutional-related PCC were the most frequently assessed in adult (86%) and paediatric (87%) populations. Within constitutional symptoms, fatigue was most frequently assessed in adult (91.6%) and paediatric (95.0%) populations, followed by fever/chills (37.9% and 55%, respectively).ConclusionsPCC definitions are heterogenous across real-world studies, which limits reliable comparisons between studies. However, some similarities were observed in terms of the most frequently measured PCC-associated symptoms/conditions, which may aid clinical management of patients with PCC.CRD42022376111.

show abstract

“… 31 Thus, a sufficient follow-up length defining people ‘at risk for long COVID’ is extremely important to minimise misclassification or misdiagnosis of PCC. 32 …”

Section: Discussionmentioning

confidence: 99%

Definition and measurement of post-COVID-19 conditions in real-world practice: a global systematic literature review

Yang,

Markus,

Andersen

et al. 2024

BMJ Open

View full text Add to dashboard Cite

show abstract

“…In a recent publication on this work, we also evaluated the capture of long COVID-19 symptoms using ICD-10 codes and natural language processing (NLP) data [ 14 ]. The Narrative Information Linear Extraction NLP tool was used to extract relevant concept unique identifiers that were manually mapped to each of the long COVID-19 symptoms we studied [ 27 ].…”

Section: Discussionmentioning

confidence: 99%

“…The use of the U09.9 code and its accuracy in identifying long COVID-19 have not yet been evaluated against any existing clinical case definitions in a multicenter setting. Clinical coding of long COVID-19 has the potential for misclassification, given the heterogeneity and ambiguity around the definition of long COVID-19 [ 14 ].…”

Section: Introductionmentioning

confidence: 99%

Characterization of Post–COVID-19 Definitions and Clinical Coding Practices: Longitudinal Study

Maripuri,

Dey,

Honerlaw

et al. 2024

Online J Public Health Inform

Self Cite

View full text Add to dashboard Cite

Background Post–COVID-19 condition (colloquially known as “long COVID-19”) characterized as postacute sequelae of SARS-CoV-2 has no universal clinical case definition. Recent efforts have focused on understanding long COVID-19 symptoms, and electronic health record (EHR) data provide a unique resource for understanding this condition. The introduction of the International Classification of Diseases, Tenth Revision (ICD-10) code U09.9 for “Post COVID-19 condition, unspecified” to identify patients with long COVID-19 has provided a method of evaluating this condition in EHRs; however, the accuracy of this code is unclear. Objective This study aimed to characterize the utility and accuracy of the U09.9 code across 3 health care systems—the Veterans Health Administration, the Beth Israel Deaconess Medical Center, and the University of Pittsburgh Medical Center—against patients identified with long COVID-19 via a chart review by operationalizing the World Health Organization (WHO) and Centers for Disease Control and Prevention (CDC) definitions. Methods Patients who were COVID-19 positive with either a U07.1 ICD-10 code or positive polymerase chain reaction test within these health care systems were identified for chart review. Among this cohort, we sampled patients based on two approaches: (1) with a U09.9 code and (2) without a U09.9 code but with a new onset long COVID-19–related ICD-10 code, which allows us to assess the sensitivity of the U09.9 code. To operationalize the long COVID-19 definition based on health agency guidelines, symptoms were grouped into a “core” cluster of 11 commonly reported symptoms among patients with long COVID-19 and an extended cluster that captured all other symptoms by disease domain. Patients having ≥2 symptoms persisting for ≥60 days that were new onset after their COVID-19 infection, with ≥1 symptom in the core cluster, were labeled as having long COVID-19 per chart review. The code’s performance was compared across 3 health care systems and across different time periods of the pandemic. Results Overall, 900 patient charts were reviewed across 3 health care systems. The prevalence of long COVID-19 among the cohort with the U09.9 ICD-10 code based on the operationalized WHO definition was between 23.2% and 62.4% across these health care systems. We also evaluated a less stringent version of the WHO definition and the CDC definition and observed an increase in the prevalence of long COVID-19 at all 3 health care systems. Conclusions This is one of the first studies to evaluate the U09.9 code against a clinical case definition for long COVID-19, as well as the first to apply this definition to EHR data using a chart review approach on a nationwide cohort across multiple health care systems. This chart review approach can be implemented at other EHR systems to further evaluate the utility and performance of the U09.9 code.

show abstract

“…While looking at the capture of PASC symptoms using ICD codes and Natural Language Processing (NLP) data, the performance of NLP is significantly better for all the commonly occurring long COVID symptoms (Figure 5). [14] The accuracy of using either a diagnosis code or NLP had the best results with 97% accuracy for loss of smell or taste and 87% accuracy for chest pain and cough.…”

Section: Discussionmentioning

confidence: 99%

“…Clinical coding of long COVID has the potential for misclassification given the heterogeneity and ambiguity around the definition of Long COVID. [14]…”

Section: Introductionmentioning

confidence: 99%

Characterization of Long COVID Definitions and Clinical Coding Practices

Maripuri,

Dey,

Honerlaw

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

BackgroundLong COVID characterized as post-acute sequelae of SARS-CoV-2 (PASC) has no universal clinical case definition. Recent efforts have focused on understanding long COVID symptoms and electronic health records (EHR) data provides a unique resource for understanding this condition. The introduction of the International Classification of Diseases (ICD)-10 code U09.9 for “Post COVID-19 condition, unspecified” to identify patients with long COVID has provided a method of evaluating this condition in EHRs, however, the accuracy of this code is unclear.ObjectiveOur study aimed to characterize the utility and accuracy of the U09.9 code across three healthcare systems - The Veterans Health Administration (VHA), Beth Israel Deaconess Medical Center (BIDMC) and The University of Pittsburgh Medical Center (UPMC) against patients identified with long COVID via a chart review by operationalizing the World Health Organization (WHO) and Centers for Disease Control (CDC) definitions.MethodsCOVID positive patients with either a U07.1 ICD code or positive polymerase chain reaction (PCR) test within these healthcare systems were identified for chart review. Among this cohort we sampled patients based on two approaches i) with a U09.9 code and ii) without a U09.9 code but with a new onset PASC related ICD code, which allows us to assess the sensitivity of the U09.9 code. To operationalize the long COVID definition based on health agency guidelines, we grouped symptoms into a “core” cluster of 11 commonly reported symptoms among long COVID patients and an extended cluster, that captured all other symptoms by disease domain.Patients having at least 2 symptoms persisting for >=60 days that were new onset after their COVID infection, with at least one symptom in the core cluster, were labeled as having long COVID per chart review. We compared the performance of the code across three health systems and across different time periods of the pandemic.ResultsA total of 900 patient charts were reviewed across 3 healthcare systems. The prevalence of long COVID among the cohort with the U09.9 ICD code, based on the operationalized WHO definition was between 23.2%-62.4% across these healthcare systems. We also evaluated a less stringent version of the WHO definition and the Centers for Disease Control (CDC) definition and observed an increase in the prevalence of long COVID at all three healthcare systems.ConclusionsThis is one of the first studies to evaluate the U09.9 code against a clinical case definition for long COVID, as well as the first to apply this definition to EHR data using a chart review approach on a nationwide cohort across multiple healthcare systems. This chart review approach can be implemented at other EHR systems to further evaluate the utility and performance of the U09.9 code.

show abstract

Potential pitfalls in the use of real-world data for studying long COVID

Cited by 17 publications

References 12 publications

Definition and measurement of post-COVID-19 conditions in real-world practice: a global systematic literature review

Definition and measurement of post-COVID-19 conditions in real-world practice: a global systematic literature review

Characterization of Post–COVID-19 Definitions and Clinical Coding Practices: Longitudinal Study

Characterization of Long COVID Definitions and Clinical Coding Practices

Contact Info

Product

Resources

About