Measuring research impact in medical research institutes: a qualitative study of the attitudes and opinions of Australian medical research institutes towards research impact assessment frameworks

Abstract

BackgroundThe question of how to measure, assess and optimise the returns from investment in health and medical research (HMR) is a highly policy-relevant issue. Research Impact Assessment Frameworks (RIAFs) provide a conceptual measurement framework to assess the impact from HMR. The aims of this study were (1) to elicit the views of Medical Research Institutes (MRIs) regarding objectives, definitions, methods, barriers, potential scope and attitudes towards RIAFs, and (2) to investigate whether an assessment framework should represent a retrospective reflection of research impact or a prospective approach integrated into the research process. The wider objective was to inform the development of a draft RIAF for Australia’s MRIs.MethodsPurposive sampling to derive a heterogeneous sample of Australian MRIs was used alongside semi-structured interviews with senior executives responsible for research translation or senior researchers affected by research impact initiatives. Thematic analysis of the interview transcriptions using the framework approach was then performed.ResultsInterviews were conducted with senior representatives from 15 MRIs. Participants understood the need for greater research translation/impact, but varied in their comprehension and implementation of RIAFs. Common concerns included the time lag to the generation of societal impacts from basic or discovery science, and whether impact reflected a narrow commercialisation agenda. Broad support emerged for the use of metrics, case study and economic methods. Support was also provided for the rationale of both standardised and customised metrics. Engendering cultural change in the approach to research translation was acknowledged as both a barrier to greater impact and a critical objective for the assessment process. Participants perceived that the existing research environment incentivised the generation of academic publications and track records, and often conflicted with the generation of wider impacts. The potential to improve the speed of translation through prospective implementation of impact assessment was supported, albeit that the mechanism required development.ConclusionThe study found that the issues raised regarding research impact assessment are less about methods and metrics, and more about the research activities that the measurement of research translation and impact may or may not incentivise. Consequently, if impact assessment is to contribute to optimisation of the health gains from the public, corporate and philanthropic investment entrusted to the institutes, then further inquiry into how the assessment process may re-align research behaviour must be prioritised.

Analysis of recent Australasian Sonographers Association (ASA) conference abstracts: How many progress to publication?

“…It is difficult to measure research impact, particularly in clinically based fields such as allied health where research success might be measured as clinical impact rather than traditional research metrics such as citation rate 33,42 . Metrics can provide a simplistic mechanism to monitor effects such as research impact by citation count, however this is not always truly reflective of impact 43 . In the academic context, journal impact factors and citations are considered a standard measure of research impact 42 .…”

“…33,42 Metrics can provide a simplistic mechanism to monitor effects such as research impact by citation count, however this is not always truly reflective of impact. 43 In the academic context, journal impact factors and citations are considered a standard measure of research impact. 42 The number of citations found for identified publications for ASA conferences was in excess of 430.…”

A science impact framework to measure impact beyond journal metrics

“…The main reason all the afore mentioned indicators are attractive is that they are quantitative and readily available. Perceptions of participants in a recent evaluation suggest that the incentivizing of publications may be at the expense of generation of broader impacts [71]. Scientific work is generally not linked to dollar investment or time to produce results.…”

Prioritising and incentivising productivity within indicator-based approaches to Research Impact Assessment: a commentary

“… Perverse incentives: generic indicators risk irrelevance across disciplines or worse, generate incentives that countermand productivity [ 74 ]. For example, bibliometric impact indicators encouraged ‘salami’ publishing [ 75 ] and academic publication indicators can incentivise revelation of intellectual property in advance of the optimal time for a commercialisation pathway [ 76 ]. In extreme cases, the established incentives can encourage misconduct by researchers [ 77 ] or institutions [ 78 ].…”