Impact occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base. Its mechanisms are complex and reflect the multiple ways in which knowledge is generated and utilised. Much progress has been made in measuring both the outcomes of research and the processes and activities through which these are achieved, though the measurement of impact is not without its critics. We review the strengths and limitations of six established approaches (Payback, Research Impact Framework, Canadian Academy of Health Sciences, monetisation, societal impact assessment, UK Research Excellence Framework) plus recently developed and largely untested ones (including metrics and electronic databases). We conclude that (1) different approaches to impact assessment are appropriate in different circumstances; (2) the most robust and sophisticated approaches are labour-intensive and not always feasible or affordable; (3) whilst most metrics tend to capture direct and proximate impacts, more indirect and diffuse elements of the research-impact link can and should be measured; and (4) research on research impact is a rapidly developing field with new methodologies on the horizon.

This report should be referenced as follows:Thompson SG, Brown LC, Sweeting MJ, Bown MJ, Kim LG, Glover MJ, et al. Systematic review and meta-analysis of the growth and rupture rates of small abdominal aortic aneurysms: implications for surveillance intervals and their cost-effectiveness. Health Technol Assess 2013;17(41). Health Technology Assessment is indexed in MEDLINE, CINAHL, EMBASE, The Cochrane Library and the ISI Science Citation Index and is assessed for inclusion in the Database of Abstracts of Reviews of Effects. Health Technology Assessment is indexed and abstracted inThis journal is a member of and subscribes to the principles of the Committee on Publication Ethics (COPE) (www.publicationethics.org/).Editorial contact: nihredit@southampton.ac.ukThe full HTA archive is freely available to view online at www.journalslibrary.nihr.ac.uk/hta. Print-on-demand copies can be purchased from the report pages of the NIHR Journals Library website: www.journalslibrary.nihr.ac.uk Criteria for inclusion in the Health Technology Assessment journalReports are published in Health Technology Assessment (HTA) if (1) they have resulted from work for the HTA programme, and (2) they are of a sufficiently high scientific quality as assessed by the reviewers and editors.Reviews in Health Technology Assessment are termed 'systematic' when the account of the search appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permit the replication of the review by others. HTA programmeThe HTA programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. 'Health technologies' are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care.The journal is indexed in NHS Evidence via its abstracts included in MEDLINE and its Technology Assessment Reports inform National Institute for Health and Care Excellence (NICE) guidance. HTA research is also an important source of evidence for National Screening Committee (NSC) policy decisions.For more information about the HTA programme please visit the website: www.hta.ac.uk/ This reportThe research reported in this issue of the journal was funded by the HTA programme as project number 08/30/02. The contractual start date was in November 2009. The draft report began editorial review in July 2012 and was accepted for publication in November 2012. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors' report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.This report presents independent research...

UK National Institute of Health Research Health Technology Assessment Programme (NIHR HTA), project number 07/32/05.

No abstract

SummaryObstructive sleep apnoea-hypopnoea (OSAH) causes excessive daytime sleepiness, impairs quality-of-life, and increases cardiovascular disease and road traffic accident risks. Continuous positive airway pressure (CPAP) treatment and mandibular advancement devices (MAD) have been shown to be effective in individual trials but their effectiveness particularly relative to disease severity is unclear.A MEDLINE, Embase and Science Citation Index search updating two systematic reviews to August 2013 identified 77 RCTs in adult OSAH patients comparing: MAD with conservative management (CM); MAD with CPAP; or CPAP with CM. Overall MAD and CPAP significantly improved apnoea-hypopnoea index (AHI) (MAD −9.3/hr (p < 0.001), CPAP −25.4 (p < 0.001)). In direct comparisons mean AHI and Epworth sleepiness scale score were lower (7.0/hr (p < 0.001) and 0.67 (p = 0.093) respectively) for CPAP. There were no CPAP vs. MAD trials in mild OSAH but in comparisons with CM, MAD and CPAP reduced ESS similarly (MAD 2.01 (p < 0.001); CPAP 1.23 (p = 0.012).Both MAD and CPAP are clinically effective in the treatment of OSAH. Although CPAP has a greater treatment effect, MAD is an appropriate treatment for patients who are intolerant of CPAP and may be comparable to CPAP in mild disease.

BackgroundImplementation of the National Health Service abdominal aortic aneurysm (AAA) screening programme (NAAASP) for men aged 65 years began in England in 2009. An important element of the evidence base supporting its introduction was the economic modelling of the long-term cost-effectiveness of screening, which was based mainly on 4-year follow-up data from the Multicentre Aneurysm Screening Study (MASS) randomized trial. Concern has been expressed about whether this conclusion of cost-effectiveness still holds, given the early performance parameters, particularly the lower prevalence of AAA observed in NAAASP.MethodsThe existing published model was adjusted and updated to reflect the current best evidence. It was recalibrated to mirror the 10-year follow-up data from MASS; the main cost parameters were re-estimated to reflect current practice; and more robust estimates of AAA growth and rupture rates from recent meta-analyses were incorporated, as were key parameters as observed in NAAASP (attendance rates, AAA prevalence and size distributions).ResultsThe revised and updated model produced estimates of the long-term incremental cost-effectiveness of £5758 (95 per cent confidence interval £4285 to £7410) per life-year gained, or £7370 (£5467 to £9443) per quality-adjusted life-year (QALY) gained.ConclusionAlthough the updated parameters, particularly the increased costs and lower AAA prevalence, have increased the cost per QALY, the latest modelling provides evidence that AAA screening as now being implemented in England is still highly cost-effective.

BackgroundThis report reviews approaches and tools for measuring the impact of research programmes, building on, and extending, a 2007 review.Objectives(1) To identify the range of theoretical models and empirical approaches for measuring the impact of health research programmes; (2) to develop a taxonomy of models and approaches; (3) to summarise the evidence on the application and use of these models; and (4) to evaluate the different options for the Health Technology Assessment (HTA) programme.Data sourcesWe searched databases including Ovid MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library from January 2005 to August 2014.Review methodsThis narrative systematic literature review comprised an update, extension and analysis/discussion. We systematically searched eight databases, supplemented by personal knowledge, in August 2014 through to March 2015.ResultsThe literature on impact assessment has much expanded. The Payback Framework, with adaptations, remains the most widely used approach. It draws on different philosophical traditions, enhancing an underlying logic model with an interpretative case study element and attention to context. Besides the logic model, other ideal type approaches included constructionist, realist, critical and performative. Most models in practice drew pragmatically on elements of several ideal types. Monetisation of impact, an increasingly popular approach, shows a high return from research but relies heavily on assumptions about the extent to which health gains depend on research. Despite usually requiring systematic reviews before funding trials, the HTA programme does not routinely examine the impact of those trials on subsequent systematic reviews. The York/Patient-Centered Outcomes Research Institute and the Grading of Recommendations Assessment, Development and Evaluation toolkits provide ways of assessing such impact, but need to be evaluated. The literature, as reviewed here, provides very few instances of a randomised trial playing a major role in stopping the use of a new technology. The few trials funded by the HTA programme that may have played such a role were outliers.DiscussionThe findings of this review support the continued use of the Payback Framework by the HTA programme. Changes in the structure of the NHS, the development of NHS England and changes in the National Institute for Health and Care Excellence’s remit pose new challenges for identifying and meeting current and future research needs. Future assessments of the impact of the HTA programme will have to take account of wider changes, especially as the Research Excellence Framework (REF), which assesses the quality of universities’ research, seems likely to continue to rely on case studies to measure impact. The HTA programme should consider how the format and selection of case studies might be improved to aid more systematic assessment. The selection of case studies, such as in the REF, but also more generally, tends to be biased towards high-impact rather than low-impact stories. Experience for other industries indicate that much can be learnt from the latter. The adoption of researchfish®(researchfish Ltd, Cambridge, UK) by most major UK research funders has implications for future assessments of impact. Although the routine capture of indexed research publications has merit, the degree to which researchfish will succeed in collecting other, non-indexed outputs and activities remains to be established.LimitationsThere were limitations in how far we could address challenges that faced us as we extended the focus beyond that of the 2007 review, and well beyond a narrow focus just on the HTA programme.ConclusionsResearch funders can benefit from continuing to monitor and evaluate the impacts of the studies they fund. They should also review the contribution of case studies and expand work on linking trials to meta-analyses and to guidelines.FundingThe National Institute for Health Research HTA programme.

SummaryBackgroundPrognosis for women with abdominal aortic aneurysm might be worse than the prognosis for men. We aimed to systematically quantify the differences in outcomes between men and women being assessed for repair of intact abdominal aortic aneurysm using data from study periods after the year 2000.MethodsIn these systematic reviews and meta-analysis, we identified studies (randomised, cohort, or cross-sectional) by searching MEDLINE, Embase, CENTRAL, and grey literature published between Jan 1, 2005, and Sept 2, 2016, for two systematic reviews and Jan 1, 2009, and Sept 2, 2016, for one systematic review. Studies were included if they were of both men and women, with data presented for each sex separately, with abdominal aortic aneurysms being assessed for aneurysm repair by either endovascular repair (EVAR) or open repair. We conducted three reviews based on whether studies reported the proportion morphologically suitable (within manufacturers' instructions for use) for EVAR (EVAR suitability review), non-intervention rates (non-intervention review), and 30-day mortality (operative mortality review) after intact aneurysm repair. Studies had to include at least 20 women (for the EVAR suitability review), 20 women (for the non-intervention review), and 50 women (for the operative mortality review). Studies were excluded if they were review articles, editorials, letters, or case reports. For the operative review, studies were also excluded if they only provided hazard ratios or only reported in-hospital mortality. We assessed the quality of the studies using the Newcastle–Ottawa scoring system, and contacted authors for the provision of additional data if needed. We combined results across studies by random-effects meta-analysis. This study is registered with PROSPERO, number CRD42016043227.FindingsFive studies assessed the morphological eligibility for EVAR (1507 men, 400 women). The overall pooled proportion of women eligible (34%) for EVAR was lower than it was in men (54%; odds ratio [OR] 0·44, 95% CI 0·32–0·62). Four single-centre studies reported non-intervention rates (1365 men, 247 women). The overall pooled non-intervention rates were higher in women (34%) than men (19%; OR 2·27, 95% CI 1·21–4·23). The review of 30-day mortality included nine studies (52 018 men, 11 076 women). The overall pooled estimate for EVAR was higher in women (2·3%) than in men (1·4%; OR 1·67, 95% CI 1·38–2·04). The overall estimate for open repair also was higher in women (5·4%) than in men (2·8%; OR 1·76, 95% CI 1·35–2·30).InterpretationCompared with men, a smaller proportion of women are eligible for EVAR, a higher proportion of women are not offered intervention, and operative mortality is much higher in women for both EVAR and open repair. The management of abdominal aortic aneurysm in women needs improvement.FundingNational Institute for Health Research (UK).