Interpretation of an online surgeon scorecard by the general public and its effect on adults changing cancer surgeons even when presented with complication rates that are statistically equivalent.

Abstract

83 Background: Recently Pro-Publica, an independent, non-profit newsroom, created a surgeon scorecard. This scorecard reports calculated death and complication rates for surgeons performing elective procedures in Medicare patients, including radical prostatectomy. We wanted to understand how the general population would interpret these data and whether it would influence surgeon choice. Methods: 265 adults at the Minnesota State Fair were recruited to participate in a survey requiring interpretation of a representative output graphic from the Pro-Publica website surgeon scorecard ( https://projects.propublica.org/surgeons ). The complication point estimate and 95% confidence intervals (CI) of a hypothetical surgeon, with whom survey participants were to assume that cancer surgery was already scheduled, were presented. They were also shown a graphic with 13 other surgeons point-estimate, complication rates, all of which fell within the index surgeon’s complication rate CI. Another surgeon, with a 0.6% lower point estimate, but statistically equivalent complication rate to the first surgeon, was indicated on the graphic. Participants were asked, based on this graphic alone, would they recommend switching surgeons? Results: The surveyed population was educated with 89% having attended, or graduated from college or higher (n = 235). Median age of participants was 50 years (Range 20-74). There were 179 females (68%) and 86 males (32%). Participants were from 136 different zip codes predominantly in the upper Midwest. 124 or 46.8% (95% CI 41-53) of respondents would recommend switching surgeons based on this single graphic. Conclusions: Nearly half of adults surveyed would recommend switching cancer surgeons for genitourinary malignancies based on data graphically presented on an online website, even though the point estimates between the two surgeons are not statically significantly different. This type of data display may lead to changes in medical decision-making based on random error of measurement of true complication rates not true differences in surgeon quality.

Optimizing Public Reporting of Congenital Heart Surgery Outcomes

“…Work by other groups highlights the potential unintended consequences of presenting information in a way that is not understandable. For example, a recent survey of adults presented with cancer surgery outcome data displayed in a fashion typical of outcome reports (designed without patient input) demonstrated that almost half of those surveyed said they would switch surgeons, when the data did not in fact show any clinically or statistically meaningful differences between the surgeons [14]. Lack of understanding of reported data has been hypothesized to be one reason why public reporting initiatives in and of themselves may not necessarily translate into improvements in overall outcomes (eg, through steering patients toward highperforming centers) in fields where reporting efforts have been in been in place for longer periods of time [15,16].…”