Application of Statistical Process Control to Physician‐specific Emergency Department Patient Satisfaction Scores: A Novel Use of the Funnel Plot

Abstract

Objectives: Emergency department (ED) patient satisfaction remains a high priority for many hospitals. Patient surveys are a common tool for measuring patient satisfaction, and process improvement efforts are aimed at improving patient satisfaction scores. In some institutions, patient satisfaction scores can be calculated for each emergency physician (EP). ED leaders are faced with the task of interpreting individual as well as group physician scores to identify opportunities for improvement. Analysis of these data can be challenging because of the relatively small numbers of returned surveys assignable to a single physician, variable numbers of surveys returned for each physician and high standard deviations (SDs) for individual physician scores. The objective was to apply statistical process control methodology to analyze individual as well as group physician patient satisfaction scores. The novel use of funnel plots to interpret individual physician patient satisfaction scores, track individual physician scores over two successive 8-month periods, and monitor physician group performance is demonstrated.Methods: Patient satisfaction with physicians was measured using Press Ganey surveys for a 65,000-volume ED over two successive 8-month periods. Using funnel plots, individual physician patient satisfaction scores were plotted against the number of surveys completed for each physician for each 8-month period. Ninety-fifth and 99th percentile control limits were displayed on the funnel plots to illustrate individual physician patient satisfaction scores that are within, versus those that are outside of, expected random variation. Control limits were calculated using mean patient satisfaction scores and SDs for the entire group of physicians. Additional funnel plots were constructed to demonstrate changes in individual physicians' patient satisfaction scores as a function of increasing numbers of returned surveys and to illustrate changes in the group's patient satisfaction scores between the first and second 8-month intervals after the institution of process improvement efforts aimed at improving patient satisfaction.Results: For the first 8-month period, 34,632 patients were evaluated in and discharged from the ED, with 581 surveys returned for 21 physicians. The mean (±SD) overall group physician patient satisfaction score was 81.8 (±24.7). Returned surveys per physician ranged from 2 to 58. For the second period, 34,858 patients were evaluated and discharged from the ED, with 670 patient satisfaction surveys returned for 20 physicians. The mean (±SD) overall physician score for all surveys returned during the second period was 85.0 (±22.2). Returned surveys per physician ranged from 8 to 65. Conclusions:The application of statistical control methodology using funnel plots as a means of analyzing ED group and physician patient satisfaction scores was possible. The authors believe that using funnel plots to analyze scores graphically can rapidly help determine the significance of individual physician p...

Guidelines on constructing funnel plots for quality indicators: A case study on mortality in intensive care unit patients

“…We recommend using exact binomial methods to construct control limits for quality indicators, which are binary at the patient level and presented as proportions, risk-adjusted rates and standardised rates. Most of the literature 6 – 17 on funnel plots lean on one study, the background of funnel plots. 3 Two studies 23 , 32 compared different methods to construct control limits and concluded that care should be taken to understand the properties of the limits constructed before using them to identify outliers; 23 and control limits obtained using probability-based prediction limits have the most logical and intuitive interpretation.…”

“…Various choices in obtaining control limits may lead to different results. Some papers describe exactly which method of this paper was used, 8 , 11 , 14 , 15 while others provide a reference but it remains unclear which method was used. 10 , 12 , 16 , 17 We found no publications describing a guideline for producing funnel plots, in which all steps required when producing a funnel plot are described.…”

“…Hence, the methods used to construct funnel plots, including obtaining control limits, need to have a solid justification in statistical theory and accepted good practice. Most published literature on comparing hospital performance [6][7][8][9][10][11][12] and registry reporting [13][14][15][16][17] using funnel plots refer to a single seminal paper on funnel plot methodology. 3 However, this paper describes multiple methods to construct control limits and it does not provide explicit guidance.…”

Clinical quality registries for clinician‐level reporting: strengths and limitations

“…Risk‐adjusted funnel plots are among the most popular methods used in presenting benchmarked clinician‐level data, 3 ‐ 5 yet they have limitations, particularly at low volumes; hence cases should still accumulate over some time before implementation of any outlier policy. Further, the true probability of falling outside of control limits varies depending on the methods used in their construction.…”

A Call to Action and Evolution

“…In the ED and often in broader pain medicine, Press Ganey ratings are virtually meaningless because the metric is flawed and inappropriately applied in these settings. Not only is the content of many patient satisfaction measures of questionable relevance, but problems with interpretation abound . As an example, excellent pain care may well entail setting boundaries and disappointing patients who had expectations of receiving opioids.…”