Assessing the accuracy of profiling methods for identifying top providers: performance of mental health care providers

Abstract

Provider profiling as a means to describe and compare the performance of health care professionals has gained momentum in the past decade. As a key component of pay-for-performance programs profiling has been increasingly used to identify top-performing providers. However, rigorous examination of the performance of statistical methods for profiling when used to classify top-performing providers is lacking. The objective of this study was to compare the classification accuracy of three methods for identifying providers exceeding performance thresholds and to analyze data on satisfaction with mental health care providers at Group Health Cooperative using these methods. Questionnaire data on patient satisfaction with mental health care providers at Group Health Cooperative was collected between April 2008 and January 2010. Simulated data were used to compare the classification accuracy of alternative statistical methods. We evaluated sensitivity, specificity, and root mean squared error of alternative statistical methods using simulated data. For Group Health providers, we compared agreement of alternative approaches to classification. We found that when between-provider variability in performance was low, all three methods exhibited poor classification accuracy. When used to evaluate mental health care provider performance, we found substantial uncertainty in the estimates and poor agreement across methods. Based on these findings, we recommend providing uncertainty estimates for provider rankings and caution against the use of any classification method when between-provider variability is low.

Provider Differences in Use of Implanted Ports in Older Adults With Cancer

“…Providers with small sample sizes are moved towards the population average, resulting in limited sensitivity to detect unusual practices. 39,40 Based on our analysis restricted to providers with 10 patients or more, we found evidence of unexplained variation between providers’ use of port. However, only 13% of all providers (having 25% of all patients assigned to them) met the threshold for inclusion, raising questions about the generalizability of these results.…”

Reliability of Physician-Level Measures of Patient Experience in Primary Care

“…38 Because patients responded to questions on a tablet device rather than a mail survey, mode effects may also influence results. 39 Although patient experience surveys are commonly used to manage physicians and to determine their compensation, 2,40 our results raise several questions about the validity and practicality of comparing individual primary care physicians based on patient experience scores. First, we found that the limited range of covariates typically adjusted for in physician-level comparisons did not account sufficiently for the variation between patients in attitudes, beliefs, and other patient characteristics that influence patients' perceptions of their healthcare experience.…”

“…After ambulatory visits, patients now commonly receive a mailed survey about their recent visit experience, and some organizations provide bonus compensation to top-performing clinicians. 2 Some have urged broader adoption of incentives based on physician-level patient experience measures and the incorporation of these measures into licensing and recertification requirements. 3 For fair comparison of individual physicians, physicianlevel patient experience measurements must have high reliability.…”

Albuminuria measurement in diabetic care: a multilevel analysis measuring the influence of accreditation on institutional performance

“…In this case, there is no need to account for patient case-mix as confounding is not a major problem by definition 35 36. The empty model, which includes only patient observations within departments, informs on the ceiling value of the department variance and its related general contextual effect 25 37. If differences in the patient measurements of albuminuria are conditioned by the general context of the level of care (ie, indicated by meaningful ICC and AUC values), then the department care is heterogeneous.…”