When to consult precision-recall curves

Cook, Jonathan; Ramadas, Vikram

doi:10.1177/1536867x20909693

Cited by 70 publications

(40 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Fifth, important measures such as those assessing interrater reliability 27 , 28 , or discrimination (c-statistic) were limited by the undefined size of the true negatives, i.e., individuals without a diagnosis code who also did not have a laboratory diagnosis is challenging to define, particularly given the variation in testing thresholds and the lack of information on the universe of patients who neither underwent testing or had a diagnosis code. The precision and recall are relevant in assessing model performance, and do not depend on this information.…”

Section: Discussionmentioning

confidence: 99%

A multicenter evaluation of computable phenotyping approaches for SARS-CoV-2 infection and COVID-19 hospitalizations

et al. 2022

View full text Add to dashboard Cite

Diagnosis codes are used to study SARS-CoV2 infections and COVID-19 hospitalizations in administrative and electronic health record (EHR) data. Using EHR data (April 2020–March 2021) at the Yale-New Haven Health System and the three hospital systems of the Mayo Clinic, computable phenotype definitions based on ICD-10 diagnosis of COVID-19 (U07.1) were evaluated against positive SARS-CoV-2 PCR or antigen tests. We included 69,423 patients at Yale and 75,748 at Mayo Clinic with either a diagnosis code or a positive SARS-CoV-2 test. The precision and recall of a COVID-19 diagnosis for a positive test were 68.8% and 83.3%, respectively, at Yale, with higher precision (95%) and lower recall (63.5%) at Mayo Clinic, varying between 59.2% in Rochester to 97.3% in Arizona. For hospitalizations with a principal COVID-19 diagnosis, 94.8% at Yale and 80.5% at Mayo Clinic had an associated positive laboratory test, with secondary diagnosis of COVID-19 identifying additional patients. These patients had a twofold higher inhospital mortality than based on principal diagnosis. Standardization of coding practices is needed before the use of diagnosis codes in clinical research and epidemiological surveillance of COVID-19.

show abstract

Section: Discussionmentioning

confidence: 99%

A multicenter evaluation of computable phenotyping approaches for SARS-CoV-2 infection and COVID-19 hospitalizations

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Nonetheless, Fawcett et al 35 advocated the use of ROC because it is insensitive to changes in the prevalence of the outcome. Cook and Ramadas 36 explained that if the primary goal, as relevant to pharmacovigilance, is to maximize sensitivity, by identifying all of the positive cases, ROC curves may still be preferable.…”

Section: Discussionmentioning

confidence: 99%

Comparison of laboratory threshold criteria in drug‐induced liver injury detection algorithms for use in pharmacovigilance

Tan

Ling

Ang

et al. 2020

Pharmacoepidemiology and Drug

View full text Add to dashboard Cite

Purpose: For the purpose of pharmacovigilance, we sought to determine the best performing laboratory threshold criteria to detect drug-induced liver injury (DILI) in the electronic medical records (EMR). Methods: We compared three commonly used liver chemistry criteria from the DILI expert working group (DEWG), DILI network (DILIN), and Council for International Organizations of Medical Sciences (CIOMS), based on hospital EMR for years 2010 and 2011 (42 176 admissions), using independent medical record review. The performance characteristics were compared in terms of sensitivity, specificity, positive predictive value (PPV), negative predictive value, accuracy, F-measure, and area under the receiver operating characteristic curve (AUROC).

show abstract

“…To assess the performance of COVID-19 diagnoses accurately identifying cases of SARS-CoV-2 infection, we assessed 3 key performance measures, precision (positive predictive value), recall (or sensitivity), and area under the precision recall curve (AUPRC). [27 28]…”

Section: Methodsmentioning

confidence: 99%

Accuracy of Computable Phenotyping Approaches for SARS-CoV-2 Infection and COVID-19 Hospitalizations from the Electronic Health Record

Khera

Mortazavi

Sangha

et al. 2021

Preprint

View full text Add to dashboard Cite

ObjectiveReal-world data, including administrative claims and electronic health record (EHR) data, have been critical for rapid-knowledge generation throughout the COVID-19 pandemic. Many studies relied on these data to identify cases and ascertain outcomes., commonly using diagnostic codes. However, to ensure high-quality results are delivered to guide clinical decision making, guide the public health response, and characterize the response to interventions, it is essential to establish the accuracy of these approaches for case identification of infections and hospitalizations.MethodsReal-world EHR data were obtained from the clinical data warehouse and computational health platform at a large academic health system that includes 5 regional hospitals in Connecticut and Rhode Island and their associated ambulatory practices. Demographic information, diagnosis codes, SARS-CoV-2 nucleic acid and antigen testing results, and visit data including discharge disposition were obtained from our OMOP common data model for all patients with either a positive SARS-CoV-2 test or ICD-10 diagnosis of COVID-19 (U07.1) between April 1, 2020 and March 1, 2021. Various computable phenotype definitions using combinations of test results and diagnostic codes were evaluated for their accuracy to identify SARS-CoV-2 infection and COVID-19 hospitalizations. The association with each phenotype was further compared with case volumes and, for hospitalizations, in-hospital mortality. We conducted a quantitative assessment with a manual chart review for a sample of 40 patients who had discordance between diagnostic code and laboratory result findings.ResultsThere were 69,423 individuals with either a diagnosis code or a laboratory diagnosis of a SARS-CoV-2 infection. Of these, 61,023 individuals had a principal or a secondary diagnosis code for COVID-19 and 50,355 had a positive SARS-CoV-2 PCR or antigen test. Among those with a positive PCR, 38,506 (76.5%) also had a principal and 3449 (6.8%) a secondary diagnosis of COVID-19, but 8400 (16.7%) had no COVID-19 diagnosis in the medical record. Moreover, of the 61,023 patients who had a COVID-19 diagnosis, 19,068 (31.2%) did not have a positive laboratory test for SARS-CoV-2 in the EHR. In a manual chart review of this sample of patients, we found that these many had a COVID-19 diagnosis code added during healthcare encounters related to asymptomatic testing, either as part of a screening program or following exposure, but with negative subsequent test results. The positive predictive value (precision) and sensitivity (recall) of a COVID-19 diagnosis in the medical record for a positive SARS-CoV-2 PCR were 68.8% and 83.3%, respectively. Further, among 5,109 patients who were hospitalized with a principal diagnosis of COVID-19, 4843 (94.8%) had a positive SARS-CoV-2 PCR or antigen test within the 2 weeks preceding hospital admission or during hospitalization. In a random sample of 10 without a positive test during the index hospitalization selected for manual chart review, 7 (70.0%) had been tested at an outside laboratory before admission and the remaining had a strong clinical suspicion for COVID-19. In addition, 789 hospitalizations had a secondary diagnosis of COVID-19, of which 446 (56.5%) had a principal diagnosis that was consistent with severe clinical manifestation of COVID-19 (e.g., sepsis or respiratory failure). Compared with the cohort that had a principal diagnosis of COVID-19, those with a secondary diagnosis more frequently male and White and had more than 2-fold higher in-hospital mortality (13.2% vs 28.0%, P<0.001).ConclusionsIn a large integrated health system, COVID-19 diagnosis codes were not adequate for case identification and epidemiological surveillance of SARS-CoV-2 infection. In contrast, a principal diagnosis of COVID-19 diagnosis consistently identified hospitalized patients with the disease but missed nearly 10% of cases that presented with more severe manifestations of disease and had over 2-fold higher mortality. Data from the EHR can provide additional data elements compared to administrative claims alone, such as laboratory testing results, that can be used to in conjunction with diagnostic codes to create more fine-tuned phenotypes that are designed for specific analytical use cases.

show abstract

When to consult precision-recall curves

Cited by 70 publications

References 18 publications

A multicenter evaluation of computable phenotyping approaches for SARS-CoV-2 infection and COVID-19 hospitalizations

A multicenter evaluation of computable phenotyping approaches for SARS-CoV-2 infection and COVID-19 hospitalizations

Comparison of laboratory threshold criteria in drug‐induced liver injury detection algorithms for use in pharmacovigilance

Accuracy of Computable Phenotyping Approaches for SARS-CoV-2 Infection and COVID-19 Hospitalizations from the Electronic Health Record

Contact Info

Product

Resources

About