Reducing ineffective practice: Challenges in identifying low-value health care using Cochrane systematic reviews

Garner, Sarah; Docherty, Mary; Somner, John; Sharma, Tarang; Choudhury, Moni; Clarke, Mike; Littlejohns, Peter

doi:10.1258/jhsrp.2012.012044

Cited by 63 publications

(78 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Academic commentators and experts in the field have suggested frameworks and models to guide disinvestment, although these remain untested. 152,155,[168][169][170] There is, however, a growing body of empirical evidence to aid in the identification of disinvestment candidates (e.g. looking at clinical practice variations).…”

Section: The Lack Of Tools and Guidance To Inform Disinvestmentmentioning

confidence: 99%

Using clinical practice variations as a method for commissioners and clinicians to identify and prioritise opportunities for disinvestment in health care: a cross-sectional study, systematic reviews and qualitative study

Hollingworth

Rooshenas

Busby

et al. 2015

Health Services and Delivery Research

View full text Add to dashboard Cite

BackgroundNHS expenditure has stagnated since the economic crisis of 2007, resulting in financial pressures. One response is for policy-makers to regulate use of existing health-care technologies and disinvest from inefficiently used health technologies. A key challenge to disinvestment is to identify existing health technologies with uncertain cost-effectiveness.ObjectivesWe aimed to explore if geographical variation in procedure rates is a marker of clinical uncertainty and might be used by local commissioners to identify procedures that are potential candidates for disinvestment. We also explore obstacles and solutions to local commissioners achieving disinvestment, and patient and clinician perspectives on regulating access to procedures.MethodsWe used Hospital Episode Statistics to measure geographical variation in procedure rates from 2007/8 to 2011/12. Expected procedure numbers for each primary care trust (PCT) were calculated adjusting for proxies of need. Random effects Poisson regression quantified the residual inter-PCT procedure rate variability. We benchmarked local procedure rates in two PCTs against national rates. We conducted rapid systematic reviews of two high-use procedures selected by the PCTs [carpal tunnel release (CTR) and laser capsulotomy], searching bibliographical databases to identify systematic reviews and randomised controlled trials (RCTs). We conducted non-participant overt observations of commissioning meetings and semistructured interviews with stakeholders about disinvestment in general and with clinicians and patients about one disinvestment case study. Transcripts were analysed thematically using constant comparison methods derived from grounded theory.ResultsThere was large inter-PCT variability in procedure rates for many common NHS procedures. Variation in procedure rates was highest where the diffusion or discontinuance was rapidly evolving and where substitute procedures were available, suggesting that variation is a proxy for clinical uncertainty about appropriate use. In both PCTs we identified procedures where high local use might represent an opportunity for disinvestment. However, there were barriers to achieving disinvestment in both procedure case studies. RCTs comparing CTR with conservative care indicated that surgery was clinically effective and cost-effective on average but provided limited evidence on patient subgroups to inform commissioning criteria and achieve savings. We found no RCTs of laser capsulotomy. The apparently high rate of capsulotomy was probably due to the coding inaccuracy; some savings might be achieved by greater use of outpatient procedures. Commissioning meetings were dominated by new funding requests. Benchmarking did not appear to be routinely carried out because of capacity issues and concerns about data reliability. Perceived barriers to disinvestment included lack of collaboration, central support and tools for disinvestment. Clinicians felt threshold criteria had little impact on their practice and that prior approval systems would not be cost-effective. Most patients were unaware of rationing.ConclusionsPolicy-makers could use geographical variation as a starting point to identify procedures where health technology reassessment or RCTs might be needed to inform policy. Commissioners can use benchmarking to identify procedures with high local use, possibly indicating overtreatment. However, coding inconsistency and limited evidence are major barriers to achieving disinvestment through benchmarking. Increased central support for commissioners to tackle disinvestment is needed, including tools, accurate data and relevant evidence. Early engagement with patients and clinicians is essential for successful local disinvestment.FundingThe National Institute for Health Research Health Services and Delivery Research programme.

show abstract

Section: The Lack Of Tools and Guidance To Inform Disinvestmentmentioning

confidence: 99%

Using clinical practice variations as a method for commissioners and clinicians to identify and prioritise opportunities for disinvestment in health care: a cross-sectional study, systematic reviews and qualitative study

Hollingworth

Rooshenas

Busby

et al. 2015

Health Services and Delivery Research

View full text Add to dashboard Cite

show abstract

“…Between them they identified 78 articles, of which 38 were deemed relevant. Of these: l 16 were descriptive or discursive 83,[301][302][303][304][305][306][307][308][309][310][311][312][313][314][315] l six were on programme budgeting and marginal analysis [316][317][318][319][320][321] l six were surveys of attitudes to disinvestment 302, [322][323][324][325][326] l five related to NICE [327][328][329][330][331] l one analysed a RCT in terms of discontinuation of a particular drug. 332 Four systematic reviews were identified, three of which were concerned with disinvestment.…”

Section: Literature Searchesmentioning

confidence: 99%

Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme

Raftery

Hanney

Greenhalgh

et al. 2016

Health Technol Assess

View full text Add to dashboard Cite

BackgroundThis report reviews approaches and tools for measuring the impact of research programmes, building on, and extending, a 2007 review.Objectives(1) To identify the range of theoretical models and empirical approaches for measuring the impact of health research programmes; (2) to develop a taxonomy of models and approaches; (3) to summarise the evidence on the application and use of these models; and (4) to evaluate the different options for the Health Technology Assessment (HTA) programme.Data sourcesWe searched databases including Ovid MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library from January 2005 to August 2014.Review methodsThis narrative systematic literature review comprised an update, extension and analysis/discussion. We systematically searched eight databases, supplemented by personal knowledge, in August 2014 through to March 2015.ResultsThe literature on impact assessment has much expanded. The Payback Framework, with adaptations, remains the most widely used approach. It draws on different philosophical traditions, enhancing an underlying logic model with an interpretative case study element and attention to context. Besides the logic model, other ideal type approaches included constructionist, realist, critical and performative. Most models in practice drew pragmatically on elements of several ideal types. Monetisation of impact, an increasingly popular approach, shows a high return from research but relies heavily on assumptions about the extent to which health gains depend on research. Despite usually requiring systematic reviews before funding trials, the HTA programme does not routinely examine the impact of those trials on subsequent systematic reviews. The York/Patient-Centered Outcomes Research Institute and the Grading of Recommendations Assessment, Development and Evaluation toolkits provide ways of assessing such impact, but need to be evaluated. The literature, as reviewed here, provides very few instances of a randomised trial playing a major role in stopping the use of a new technology. The few trials funded by the HTA programme that may have played such a role were outliers.DiscussionThe findings of this review support the continued use of the Payback Framework by the HTA programme. Changes in the structure of the NHS, the development of NHS England and changes in the National Institute for Health and Care Excellence’s remit pose new challenges for identifying and meeting current and future research needs. Future assessments of the impact of the HTA programme will have to take account of wider changes, especially as the Research Excellence Framework (REF), which assesses the quality of universities’ research, seems likely to continue to rely on case studies to measure impact. The HTA programme should consider how the format and selection of case studies might be improved to aid more systematic assessment. The selection of case studies, such as in the REF, but also more generally, tends to be biased towards high-impact rather than low-impact stories. Experience for other industries indicate that much can be learnt from the latter. The adoption of researchfish®(researchfish Ltd, Cambridge, UK) by most major UK research funders has implications for future assessments of impact. Although the routine capture of indexed research publications has merit, the degree to which researchfish will succeed in collecting other, non-indexed outputs and activities remains to be established.LimitationsThere were limitations in how far we could address challenges that faced us as we extended the focus beyond that of the 2007 review, and well beyond a narrow focus just on the HTA programme.ConclusionsResearch funders can benefit from continuing to monitor and evaluate the impacts of the studies they fund. They should also review the contribution of case studies and expand work on linking trials to meta-analyses and to guidelines.FundingThe National Institute for Health Research HTA programme.

show abstract

“…55 The named contact revealed that they were about to start heading up a project group representing a DOI: 10.3310/hsdr05050 HEALTH SERVICES AND DELIVERY RESEARCH 2017 VOL. 5 NO.…”

Section: Instrumental Use Of Research Evidence: Low-value Interventionsmentioning

confidence: 99%

“…Despite this, practical challenges in identifying and contextualising research evidence to inform these processes remain. 55 Unlike the rigorous processes in place to inform the NICE guidance on the use of new and existing medicines, no similarly resourced infrastructure exists to support disinvestment decisions. 7 Although NICE makes 'do not do' recommendations publicly available, we found low awareness of these among commissioners and a notable lack of skills to systematically and transparently identify other relevant evidence that could inform disinvestment decisions.…”

Section: Reflections On Deliverymentioning

confidence: 99%

Effects of a demand-led evidence briefing service on the uptake and use of research evidence by commissioners of health services: a controlled before-and-after study

Wilson

Farley

Bickerdike

et al. 2017

Health Serv Deliv Res

View full text Add to dashboard Cite

BackgroundThe Health and Social Care Act 2012 (Great Britain.Health and Social Care Act 2012. London: The Stationery Office; 2012) has mandated research use as a core consideration of health service commissioning arrangements. We evaluated whether or not access to a demand-led evidence briefing service improved the use of research evidence by commissioners, compared with less intensive and less targeted alternatives.DesignControlled before-and-after study.SettingClinical Commissioning Groups (CCGs) in the north of England.Main outcome measuresChange at 12 months from baseline of a CCG’s ability to acquire, assess, adapt and apply research evidence to support decision-making. Secondary outcomes measured individual clinical leads’ and managers’ intentions to use research evidence in decision-making.MethodsNine CCGs received one of three interventions: (1) access to an evidence briefing service; (2) contact plus an unsolicited push of non-tailored evidence; or (3) an unsolicited push of non-tailored evidence. Data for the primary outcome measure were collected at baseline and 12 months post intervention, using a survey instrument devised to assess an organisation’s ability to acquire, assess, adapt and apply research evidence to support decision-making. In addition, documentary and observational evidence of the use of the outputs of the service was sought and interviews with CCG participants were undertaken.ResultsMost of the requests were conceptual; they were not directly linked to discrete decisions or actions but were intended to provide knowledge about possible options for future actions. Symbolic use to justify existing decisions and actions were less frequent and included a decision to close a walk-in centre and to lend weight to a major initiative to promote self-care already under way. The opportunity to impact directly on decision-making processes was limited to work to establish disinvestment policies. In terms of impact overall, the evidence briefing service was not associated with increases in CCGs’ capacity to acquire, assess, adapt and apply research evidence to support decision-making, individual intentions to use research findings or perceptions of CCGs’ relationships with researchers. Regardless of the intervention received, at baseline participating CCGs indicated that they felt that they were inconsistent in their research-seeking behaviours and their capacity to acquire research remained so at follow-up. The informal nature of decision-making processes meant that there was little or no traceability of the use of evidence.LimitationsLow baseline and follow-up response rates (of 68% and 44%, respectively) and missing data limit the reliability of these findings.ConclusionsAccess to a demand-led evidence briefing service did not improve the uptake and use of research evidence by NHS commissioners compared with less intensive and less targeted alternatives. Commissioners appear to be well intentioned but ad hoc users of research.Future workFurther research is required on the effects of interventions and strategies to build individual and organisational capacity to use research. Resource-intensive approaches to providing evidence may best be employed to support instrumental decision-making. Comparative evaluation of the impact of less intensive but targeted strategies on the uptake and use of research by commissioners is warranted.FundingThe National Institute for Health Research Health Services and Delivery Research programme.

show abstract

Reducing ineffective practice: Challenges in identifying low-value health care using Cochrane systematic reviews

Cited by 63 publications

References 22 publications

Using clinical practice variations as a method for commissioners and clinicians to identify and prioritise opportunities for disinvestment in health care: a cross-sectional study, systematic reviews and qualitative study

Using clinical practice variations as a method for commissioners and clinicians to identify and prioritise opportunities for disinvestment in health care: a cross-sectional study, systematic reviews and qualitative study

Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme

Effects of a demand-led evidence briefing service on the uptake and use of research evidence by commissioners of health services: a controlled before-and-after study

Contact Info

Product

Resources

About