The impact of the National Institute for Health Research Health Technology Assessment programme, 2003–13: a multimethod evaluation

Abstract

BackgroundThe National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme supports research tailored to the needs of NHS decision-makers, patients and clinicians. This study reviewed the impact of the programme, from 2003 to 2013, on health, clinical practice, health policy, the economy and academia. It also considered how HTA could maintain and increase its impact.MethodsInterviews (n = 20): senior stakeholders from academia, policy-making organisations and the HTA programme. Bibliometric analysis: citation analysis of publications arising from HTA programme-funded research. Researchfish survey: electronic survey of all HTA grant holders. Payback case studies (n = 12): in-depth case studies of HTA programme-funded research.ResultsWe make the following observations about the impact, and routes to impact, of the HTA programme: it has had an impact on patients, primarily through changes in guidelines, but also directly (e.g. changing clinical practice); it has had an impact on UK health policy, through providing high-quality scientific evidence – its close relationships with the National Institute for Health and Care Excellence (NICE) and the National Screening Committee (NSC) contributed to the observed impact on health policy, although in some instances other organisations may better facilitate impact; HTA research is used outside the UK by other HTA organisations and systematic reviewers – the programme has an impact on HTA practice internationally as a leader in HTA research methods and the funding of HTA research; the work of the programme is of high academic quality – theHealth Technology Assessmentjournal ensures that the vast majority of HTA programme-funded research is published in full, while the HTA programme still encourages publication in other peer-reviewed journals; academics agree that the programme has played an important role in building and retaining HTA research capacity in the UK; the HTA programme has played a role in increasing the focus on effectiveness and cost-effectiveness in medicine – it has also contributed to increasingly positive attitudes towards HTA research both within the research community and the NHS; and the HTA focuses resources on research that is of value to patients and the UK NHS, which would not otherwise be funded (e.g. where there is no commercial incentive to undertake research). The programme should consider the following to maintain and increase its impact: providing targeted support for dissemination, focusing resources when important results are unlikely to be implemented by other stakeholders, particularly when findings challenge vested interests; maintaining close relationships with NICE and the NSC, but also considering other potential users of HTA research; maintaining flexibility and good relationships with researchers, giving particular consideration to the Technology Assessment Report (TAR) programme and the potential for learning between TAR centres; maintaining the academic quality of the work and the focus on NHS need; considering funding research on the short-term costs of the implementation of new health technologies; improving the monitoring and evaluation of whether or not patient and public involvement influences research; improve the transparency of the priority-setting process; and continuing to monitor the impact and value of the programme to inform its future scientific and administrative development.FundingThe NIHR HTA programme.

The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews

“…Various examples of the latter were identified in the two reviews [73–79]. One study, that of Johnston et al [49], occupies a particular place in the consideration of frameworks because it included all individual projects within a programme (see above) and, while all of the projects were examined, only a small proportion were identified as making a measurable impact.…”

“…Some other studies with a more limited scope have also used a bottom-up approach to assess the impact of specific projects, but have not gone as far as attempting a comprehensive valuation of the impact of a whole programme of research. Nevertheless, such studies can indicate probable minimum levels of returns from the whole programme studied [79]. …”

Evidence synthesis on measuring the distribution of benefits of research and innovation

“…Oortwijn et al 2008;McClure et al 2012;Scott et al 2011;Wooding et al 2009). An example of this is the two studies evaluating the NIHR's Health Technology Assessment (HTA) programme, which drew on methods including bibliometric analysis, interviews, surveys and case studies to characterise the outcomes of the research funded through the programme, from the initial production of knowledge to ultimate impacts on health and wellbeing (Guthrie et al 2015;Hanney et al 2007). This type of mixed-methods approach is fairly typical in evaluations of programmes of health and biomedical research, in that it attempts to capture evidence along the pathway from knowledge production to changes in policy and practice, and to ultimate health benefits, drawing on a combination of methods and metrics.…”

Mechanisms and pathways to impact in public health research: a preliminary analysis of research funded by the National Institute for Health Research (NIHR)

“…There is also evidence suggesting that some impacts are achieved during the research process itself, especially in cases where the research was co-created with beneficiaries. Studies such as the Retrosight studies [13,14], the analysis of the REF 2014 impact case studies [15], and a recent evaluation of the NIHR HTA programme [16], have all highlighted examples of research where study participants had already received the benefits of the research. Analyses of impact therefore often include the intermediary steps, or 'proxies' to the ultimate impact.…”