Truth in Reporting: How Data Capture Methods Obfuscate Actual Surgical Site Infection Rates within a Health Care Network System

Abstract

Background Two systems measure surgical site infection rates following colorectal surgeries. Center for Medicare and Medicaid Services pay-for-performance initiatives use National Healthcare Safety Network data for hospital comparisons. Objective Compare database concordance. Design Multi-institution cohort study of system-wide Colorectal Surgery Collaborative. The National Surgical Quality Improvement Program requires rigorous, standardized data capture techniques; National Healthcare Safety Network allows five data capture techniques. Standardized surgical site infection rates were compared between databases. Cohen's Kappa coefficient calculated. Setting Boston-area hospitals. Patients National Healthcare Safety Network or National Surgical Quality Improvement Program patients undergoing colorectal surgery. Main Outcome Measures Standardized surgical-site infection rates Results Thirty-day surgical-site infection rates of 3,547 (National Surgical Quality Improvement Program) vs 5,179 (National Healthcare Safety Network) colorectal procedures (2012-2014). Discrepancies appeared: National Surgical Quality Improvement Program database of Hospital 1 (N=1,480 patients) routinely found surgical-site infection rates around 10%, routinely deemed rate “exemplary” or “as expected” (100%). National Healthcare Safety Network data from the same hospital and time period (N=1,881) revealed similar overall surgical-site infection rate (10%), but standardized rates were deemed “worse than national average” 80% of the time. Overall, hospitals using less rigorous capture methods had improved surgical-site infection rates for National Healthcare Safety Network compared to standardized National Surgical Quality Improvement Program reports. The correlation coefficient between standardized infection rates was 0.03 (p=0.88). During 25 site-time period observations, National Surgical Quality Improvement Program and National Healthcare Safety Network data matched for 52% of observations (13/25). Kappa=0.10 (95% CI: -0.1366-0.3402; p=0.403), indicating poor agreement. Limitations Hospitals located in Northeastern United States only. Conclusions Variation in Center for Medicare and Medicaid Services-mandated National Healthcare Safety Network infection surveillance methodology leads to unreliable results, which is apparent when these results are compared to standardized data. High quality data would improve care quality and compare outcomes amongst institutions.

Surgical site infection metrics: Dissecting the differences between the National Health and Safety Network and the National Surgical Quality Improvement Program

“…The one metric collected by both surveillance systems is SSI. We noticed that at our facility, as has been reported by others, [1][2][3][4][5] NHSN and NSQIP metric reports have often painted conflicting pictures of our hospital's SSI performance as benchmarked against our peers. This discordance has raised questions about the validity of the SSI metric and has threatened to undermine the original intent of the surveillance endeavor, which is to improve care.…”

“…Surveillance for the purpose of reporting to the NHSN involves a process that varies from hospital to hospital, with general instructions outlined by the CDC in the NHSN Patient Safety Component Manual. 4 For example, at our institution, infection preventionists monitor admission and readmission lists, lists of patients on contact precautions, all positive microbiology results, and morbidity and mortality conference case lists. Inherent to this inpatientfocused methodology is a lack of resources to detect SSI events that are diagnosed at outpatient visits or at other facilities; this occurs most often in the case of superficial incisional SSIs, which may not warrant culture, readmission, or reoperation.…”

Framework to Evaluate Level of Good Faith in Implementations of Public Dashboards

“…Since catheterization happens in the intensive care unit (ICU) setting, typically hospitals track CAUTI and CLABSI as part of healthcare quality activities centered around ICUs [14]. By contrast, SSIs are tracked in association with specific operative procedures (e.g., colon surgery) [24].…”

“…In the US compared to other countries, rates of HAI are relatively high, likely because they are not required to be tracked at the federal level [14,25]. Hospitals can opt into the federal voluntary tracking system called the National Health Safety Network (NHSN), but the NHSN does not have a publicly-facing dashboard, and the data are inaccurate, especially in undercounting severe events [14,24,26]. As HAI is a serious public health issue, there has been a call for greater data transparency, so the reports posted on the MA DPH web site represent MA's attempt to comply with state-level mandates for OGD.…”

Surveillance quality correlates with surgical site infection rates in knee and hip arthroplasty and colorectal surgeries: A call to action to adjust reporting of SSI rates

“…[10][11][12] However, the quality of surveillance itself has also come under scrutiny, in particular in the age of increased automation (eg, Pindyck et al 13 and van Mourik et al 14 ), and, at least in some countries, linkage of financial penalties with SSI rates. In their 2017 article, Bordeianou et al 15 point out that "overall, hospitals using less rigorous capture methods had improved surgical-site infection rates for National Healthcare Safety Network [NHSN] compared to standardized National Surgical Quality Improvement Program reports," and "studies have found large discrepancies in the SSI rates reported in NHSN data compared to internal audits." In Switzerland, repetitive audits of the surveillance processes and outcomes were introduced in 2011, with many hospitals in the Swissnoso surveillance program having 3 audits (in our analysis 61 of 148 hospitals, 41%) in the period up to the end of 2019.…”