Measuring research impact in Australia’s medical research institutes: a scoping literature review of the objectives for and an assessment of the capabilities of research impact assessment frameworks

Abstract

BackgroundRealising the economic potential of research institutions, including medical research institutes, represents a policy imperative for many Organisation for Economic Co-operation and Development nations. The assessment of research impact has consequently drawn increasing attention. Research impact assessment frameworks (RIAFs) provide a structure to assess research translation, but minimal research has examined whether alternative RIAFs realise the intended policy outcomes. This paper examines the objectives presented for RIAFs in light of economic imperatives to justify ongoing support for health and medical research investment, leverage productivity via commercialisation and outcome–efficiency gains in health systems, and ensure that translation and impact considerations are embedded into the research process.This paper sought to list the stated objectives for RIAFs, to identify existing frameworks and to evaluate whether the identified frameworks possessed the capabilities necessary to address the specified objectives.MethodsA scoping review of the literature to identify objectives specified for RIAFs, inform upon descriptive criteria for each objective and identify existing RIAFs. Criteria were derived for each objective. The capability for the existing RIAFs to realise the alternative objectives was evaluated based upon these criteria.ResultsThe collated objectives for RIAFs included accountability (top-down), transparency/accountability (bottom-up), advocacy, steering, value for money, management/learning and feedback/allocation, prospective orientation, and speed of translation. Of the 25 RIAFs identified, most satisfied objectives such as accountability and advocacy, which are largely sufficient for the first economic imperative to justify research investment. The frameworks primarily designed to optimise the speed of translation or enable the prospective orientation of research possessed qualities most likely to optimise the productive outcomes from research. However, the results show that few frameworks met the criteria for these objectives.ConclusionIt is imperative that the objective(s) for an assessment framework are explicit and that RIAFs are designed to realise these objectives. If the objectives include the capability to pro-actively drive productive research impacts, the potential for prospective orientation and a focus upon the speed of translation merits prioritisation. Frameworks designed to optimise research translation and impact, rather than simply assess impact, offer greater promise to contribute to the economic imperatives compelling their implementation.

Reporting health research translation and impact in the curriculum vitae: a survey

“…Health researchers are increasingly required to show evidence of translation of their research findings to demonstrate real-world impact [9,24]. As such, it is important that health researchers capitalize on opportunities to document and communicate the impact of their research.…”

Assessing Health Research and Innovation Impact: Evolution of a Framework and Tools in Alberta, Canada

“…Funders and other organizations are motivated to assess the impact of their investments for several reasons. A review of 25 research impact assessment frameworks by Deeming et al (2017) found that frameworks are designed for different objectives: Accountability for research impact at the sector, state, and national level of governments was found to be the most frequent objective of the reviewed frameworks (Deeming et al, 2017). Closely aligned with and complementary to accountability was transparency because the collection of information about research activities, outputs and outcomes serves as a "bottom up" accountability measure along the pathways to impact.…”

“…Integrating the work of Deeming et al (2017) with the teachings of ISRIA, brief descriptions of a sample of six research impact assessment frameworks are provided in Table 1. Table 1 considers advantages and disadvantages of each framework and highlights the need to articulate the assessment purpose and consider the different types of methods and data collection tools that will be used to meet the purpose.…”

Advancing the management of obstructive airways diseases through translational research

“…These challenges are, however, not reason to abandon efforts to measure research impact. Measurement can be a powerful motivator: when set in a prospective framework that allows for feedback, it can be a means to encourage research behaviours and activities that are associated with an increased likelihood of research translation and research impact . Therefore, part of the solution is to use an impact measurement framework that incorporates process metrics that encourage known translational activities and behaviours.…”

“…The body of knowledge as to what these activities and behaviours might be is rapidly developing . As an example, successful translation has been associated with early engagement with end‐users, the development of a strategic plan explaining the translational pathway (eg a program logic model) and an understanding of the barriers to translation …”