What is a performance outlier?

Abstract

To cite: Shahian DM, Normand S-LT. BMJ Qual Saf 2015;24:95-99. Healthcare performance measurement is a complex undertaking, often presenting a number of potential alternative approaches and methodological nuances. Important considerations include richness and quality of data sources; data completeness; choice of metrics and target population; sample size; patient-and provider-level data collection periods; risk adjustment; statistical methodology (eg, logistic regression vs hierarchical models); model performance, reliability and validity; and classification of outliers. Given these many considerations, as well as the absence of nationally accepted standards for provider profiling, it is not surprising that different rating organisations and methodologies may produce divergent results for the same hospitals.

Congenital Heart Surgery Case Mix Across North American Centers and Impact on Performance Assessment

“…Healthcare performance measurement is a complex undertaking, and involves several important considerations, including: the data source, choice of performance metrics, target population, sample size/power, adjustment for patient risk-factors and differences in procedural case-mix, statistical methodology, classification of performance and outliers, and interpretation of indirectly standardized outcomes (5,11). The present study focused primarily on the aspects of case-mix and target population.…”

“…This is because it can be challenging for any type of adjustment (e.g. model-based extrapolation) to reliably account for the lack of data in an area of non-overlap in case-mix between centers (5,11). For example, in adult cardiac surgery, the STS reports center performance separately by operation type, such as isolated coronary artery bypass grafting (CABG), isolated aortic valve replacement, combined CABG/aortic valve replacement, etc (4).…”

Deaths by horsekick in the Prussian army - and other ‘Never Events’ in large organisations

“…Different analyses can yield opposite results, where hospitals identified as poor performers by one method are statistically no different from a hospital marked as a high performer. Problems of interpretation can arise where sample size is inevitably low (as it is with Never Events) and confidence intervals can be wide [16][17][18].…”

Unpacking quality indicators: how much do they reflect differences in the quality of care?

“…Determining the effectiveness of healthcare performance measurement is challenging,11 particularly at the health system level. Often, performance measurement is implemented uniformly across an entire system, making well-designed controlled analysis less feasible or impossible12 13 and leaving evaluations vulnerable to secular trends 14.…”