The Measurement Reliability of Complications and Patient Satisfaction After Common Surgical Procedures

Abstract

Objective: To evaluate the reliability of surgeon outcomes. Background: Surgeons’ outcomes are now widely used in public reporting and value-based reimbursement, but the reliability of these measures continues to raise concerns. Methods: We performed a retrospective study of surgeons performing cholecystectomy, colectomy, and hernia repair on adult patients between January 1, 2017, and December 31, 2020. Outcomes were risk-adjusted rates of complications and high patient satisfaction. We estimated the reliability of each outcome, its relationship with case volume, and the number of surgeons who reached an acceptable level of reliability (≥0.70). Results: A total of 23,533 patients with a mean age of 56.8 (16.2) years and 10,191 (43.3%) females underwent operations by 333 surgeons. Risk-adjusted complication rate was 2.5% [95% confidence interval (CI): 2.2%–2.8%] and risk-adjusted high satisfaction rate was 79.9% (95% CI: 78.7%–81.0%). The reliability of the complication rate was 0.27 (95% CI: 0.25–0.29) and the reliability of the high satisfaction rate was 0.53 (95% CI: 0.50–0.55). Reliability increased with case volume; however, only 5 (1.5%) surgeons performed enough cases to reach acceptable reliability for their complication rate, while 86 (25.8%) surgeons reached acceptable reliability for their patient satisfaction rate. After adjustment for reliability, the range of complication rates decreased 29-fold from 0% to 14.3% to 2.4% to 2.9%, and the range of patient satisfaction decreased 2.6-fold from 25.3% to 100.0% to 64.9% to 92.4%. Conclusions: Among surgeons performing common operations, complications and patient satisfaction had relatively low reliability. Although reliability increased with volume, most surgeons had insufficient case volume to achieve acceptable reliability of their outcomes. As such, these measures likely offer little to no meaningful information to inform decision-making.