Reliability of Surgeon-specific Reporting of Complications After Colectomy

Abstract

Objective We sought to determine the reliability of surgeon-specific postoperative complication rates after colectomy. Background Conventional measures of surgeon-specific performance fail to acknowledge variation attributed to statistical noise, risking unreliable assessment of quality. Methods We examined all patients who underwent segmental colectomy with anastomosis from 2008 through 2010 participating in the Michigan Surgical Quality Collaborative (MSQC) Colectomy Project. Surgeon-specific complication rates were risk-adjusted according to patient characteristics with multiple logistic regression. Hierarchical modeling techniques were used to determine the reliability of surgeon-specific risk-adjusted complication rates. We then adjusted these rates for reliability. To evaluate the extent to which surgeon-level variation was reduced, surgeons were placed into quartiles based on performance and complication rates were compared before and after reliability adjustment Results A total of 5,033 patients (n=345 surgeons) undergoing partial colectomy reported a risk-adjusted complication rate of 24.5%. Approximately 86% of the variability of complication rates across surgeons was explained by measurement noise, while the remaining 14% represented true signal. Risk-adjusted complication rates varied from 0% to 55.1% across quartiles prior to adjusting for reliability. Reliability adjustment greatly diminished this variation, generating a 1.2 fold difference (21.4%-25.6%). A caseload of 168 colectomies across three years was required to achieve a reliability of >0.7, which is considered a proficient level. Only one surgeon surpassed this volume threshold. Conclusions The vast majority of surgeons do not perform enough colectomies to generate a reliable surgeon-specific complication rate. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume, as statistical noise is a large determinant in estimating their surgeon-specific complication rates.

Anastomotic leak after colorectal resection: A population-based study of risk factors and hospital variation

“…Finally, the choice to perform provider comparisons at the hospital level rather than the surgeon level was related to the inability to make statistically valid comparisons at the surgeon level due to (1) our agreement for use of the data, which did not allow for surgeon identifiers, and (2) small case volumes at the surgeon level. 14 Nevertheless, future work at the surgeon level may be possible (as the volume of available cases with this outcome increases with additional data collection).…”

“…10–13 Until more generalizable studies are available which enumerate the most important risk-factors, 8 performing risk-adjusted comparisons between providers is not possible. 14 In addition, many studies utilize administrative data which rely on surrogates that may not capture ALs effectively. 15 The demonstration of variation in risk-adjusted rates of clinically defined AL between providers would be an important finding, indicating the potential for performance improvement.…”

Surgeon‐specific outcome reporting: is it time to move forward?

“…Intra-and postoperative death is, however, very uncommon in elective colorectal surgery and this limits the value of assessing the statistical significance of one surgeon's practice. Shih et al [8] found that most surgeons do not perform enough colectomies (less than 56/year) to generate a statistically reliable surgeon-specific complication rate and concluded that underpowered results are unreliable.…”

Creating Individual Surgeon Performance Assessments in a Statewide Hospital Surgical Quality Improvement Collaborative

“…Researchers have long attempted to profile provider performance, targeting different types of providers using various measures, with limited success. [1][2][3][4][5][6][7][8] Surgeons have been the focus of numerous profiling efforts focused on postoperative patient outcomes, dating back to the New York Department of Health's publication of mortality rates after coronary artery bypass grafting surgery in the early 1990s. 9 In 2013, the United Kingdom's National Health Service began publishing surgeon-specific patient outcomes.…”