Are Improvements in Measured Performance Driven by Better Treatment or “Denominator Management”?

Abstract

BACKGROUND: Process measures of healthcare quality are usually formulated as the number of patients who receive evidence-based treatment (numerator) divided by the number of patients in the target population (denominator). When the systems being evaluated can influence which patients are included in the denominator, it is reasonable to wonder if improvements in measured quality are driven by expanding numerators or contracting denominators. OBJECTIVE: In 2003, the US Department of Veteran Affairs (VA) based executive compensation in part on performance on a substance use disorder (SUD) continuity-ofcare quality measure. The first goal of this study was to evaluate if implementing the measure in this way resulted in expected improvements in measured performance. The second goal was to examine if the proportion of patients with SUD who qualified for the denominator contracted after the quality measure was implemented, and to describe the facility-level variation in and correlates of denominator contraction or expansion. DESIGN: Using 40 quarters of data straddling the implementation of the performance measure, an interrupted time series design was used to evaluate changes in two outcomes. PARTICIPANTS: All veterans with an SUD diagnosis in all VA facilities from fiscal year 2000 to 2009. MAIN MEASURES: The two outcomes were 1) measured performance-patients retained/patients qualified and 2) denominator prevalence-patients qualified/patients with SUD program contact. KEY RESULTS: Measured performance improved over time (P < 0.001). Notably, the proportion of patients with SUD program contact who qualified for the denominator decreased more rapidly after the measure was implemented (p = 0.02). Facilities with higher pre-implementation denominator prevalence had steeper declines in denominator prevalence after implementation (p < 0.001). CONCLUSIONS: These results should motivate the development of measures that are less vulnerable to denominator management, and also the exploration of Bshadow measures^to monitor and reduce undesirable denominator management.

Effects of Expanding Health Screening on Treatment - What Should We Expect? What Can We Learn?

“…Third, our research contributes to the small but growing literature on the unintended effects of quality reporting (Casalino et al, 2007;Dranove et al, 2003;Harris et al, 2016;Karve et al, 2008). In multiple pay-for-performance systems such as Accountable Care Organizations, providers have financial incentives to maintain high treatment rates for diagnosed conditions as well as high screening rates (Center for Medicare and Medicaid Services, 2011, 2016a; National Committee on Quality Assurance, 2016; Song et al, 2011Song et al, , 2014.…”

The Next Generation of Clinical Performance Measures

“…Using data for veterans with SUD who were treated in VA facilities from FY 2000 to FY 2009, Harris and colleagues evaluated whether implementing the measure resulted in expected improvements in performance. 7 Findings show that including an SUD continuity-of-care quality measure in network directors' performance contracts was associated with an increase in measured performance, from 23 % just before the measure was implemented to 48 % by the end of the observation period. At the same time, the overall proportion of patients with SUD program contact that qualified for the measure decreased more rapidly over time following implementation, and varied significantly among facilities.…”

Pay-for-Performance and Veteran Care in the VHA and the Community: a Systematic Review

“…Effectiveness of P4P. Four articles 13,23,29,30 from three studies 13,23,29 provide data on the effectiveness of P4P in VHA settings (see online Appendix 7 for detail). Overall, the evidence is insufficient to determine whether P4P results in durable improvements in the quality of care or health of Veterans.…”

“…However, it is possible that these findings may have been influenced by concomitant public reporting 23 and denominator management (i.e., a decrease in the number of patients eligible for a performance measure that may be positive, resulting in improvements in identification; or negative, resulting from gaming). 29 Unintended Consequences. Eleven studies published in 13 articles 13-15, 24, 29-36, 42 examined potential unintended consequences in VHA settings (see online Appendices 8 and 9).…”

“…14,31,32,34 However, one RCT found that P4P for hypertension did not increase the risk of hypotension despite findings from a sub-study that suggested subjects were concerned about the risk of overtreatment. 13,42 Other studies examining unintended consequences reported findings congruent with denominator management, 29 but no evidence of risk selection. 30,33 Qualitative studies found that participants felt performance measures may lead to negative unintended consequences such as reduced focus on patient needs/concerns, unincentivized areas of care, and/or healthier patient populations (teaching to the test/attention shift), 15,24,35 and that they may negatively affect team dynamics, particularly if metrics are incentivized.…”