Investigating the nature and quality of locally commissioned evaluations of the NHS Vanguard programme: an evidence synthesis

Abstract

Background With innovation in service delivery increasingly viewed as crucial to the long-term sustainability of health systems, NHS England launched an ambitious new model of care (Vanguard) programme in 2015. Supported by a £350 million transformation fund, 50 Vanguard sites were to act as pilots for innovation in service delivery, to move quickly to change the way that services were delivered, breaking down barriers between sectors and improving the coordination and delivery of care. Methods As part of a national evaluation of the Vanguard programme, we conducted an evidence synthesis to assess the nature and quality of locally commissioned evaluations. With access to a secure, online hub used by the Vanguard and other integrated care initiatives, two researchers retrieved any documents from a locally commissioned evaluation for inclusion. All identified documents were downloaded and logged, and details of the evaluators, questions, methodological approaches and limitations in design and/or reporting were extracted. As included evaluations varied in nature and type, a narrative synthesis was undertaken. Results We identified a total of 115 separate reports relating to the locally commissioned evaluations. Five prominent issues relating to evaluation conduct were identified across included reports: use of logic models, number and type of evaluation questions posed, data sharing and information governance, methodological challenges and evaluation reporting in general. A combination of resource, data and time constraints means that evaluations often attempted to but did not fully address the wide range of questions posed by individual Vanguards. Conclusions Significant investment was made in independent local evaluations of the Vanguard programme by NHS England. This synthesis represents the only comprehensive attempt to capture methodological learning and may serve as a key resource for researchers and policy-makers seeking to understand investigating large-scale system change, both within the NHS and internationally. PROSPERO (Registration number: CRD42017069282).

Comparison of the impact of two national health and social care integration programmes on emergency hospital admissions

“…Moreover, implemented initiatives were heterogeneous across sites and difficult to describe in detail using interviews and documentary analysis [15,19,[32][33][34][35][36][37]. A recent evidence synthesis of 115 local Vanguard evaluation reports concluded that none of the reports offered "explanations and/or nuanced insights into the Vanguard operation" [38], making crossinitiatives comparisons difficult. Second, our method does not rule out the possibility that confounding events, unrelated to the integration processes, may have differentially affected the different groups of CCGs being compared in the post-implementation period.…”

“…Hitherto, the Pioneer or Vanguard programmes have been studied separately [15,19,[32][33][34][35][36][37][38], but never in combination. However, a number of Pioneers subsequently applied to become Vanguards, and thus the two programmes overlapped in time and partially in place.…”

Scale and spread of innovation in health and social care: Insights from the evaluation of the New Care Model/Vanguard programme in England

“…Substantial funding was an important influence on the implementation of the Vanguard pilots. 25 However, although funding was effectively ring-fenced around Vanguards, no This meant that many Vanguard initiatives were terminated 19 impacting future initiatives as highlighted by this service commissioner:…”

“…Although the local U n d e r R e v i e w 17 evaluation teams fed back interim findings, this was often either too late for Vanguards to change course or there was a desire, and associated pressure, to demonstrate success against nationally determined parameters. 25 This was reflected in tension between some of the Vanguards and their evaluators, where the Vanguards saw the evaluators' role as confirming success rather than using the feedback as a mechanism to learn and refine their activities, as this local evaluator said: 'They thought they were paying us to give us positive results but in order to maintain our integrity we had to be as neutral as possible.' (P1R11) Increasingly, the Vanguards came under pressure to demonstrate success against nationally determined outcome measures that did not necessarily align with local objectives.…”

Vertical integration of GP practices with acute hospitals in England and Wales: rapid evaluation

“…20 However, data and findings are limited across the vanguard sites, with questions about the reliability of outcomes data and, consequently, uncertainty about the impact of PACS on key dimensions of health service delivery. 24,25 Five years on from the NHS Five Year Forward View, 21 the 2019 NHS Long Term Plan 26 announced the intention that all general practices in England should combine into PCNs covering populations of 30,000-50,000. 26 Since July 2019, all but a small number of practices have become horizontally integrated in that way with other practices, although remaining separate legal entities with separate contracts.…”