‘Making evidence count’: A framework to monitor the impact of health services research

Buykx, Penny; Humphreys, John Stirling; Wakerman, John; Perkins, David; Lyle, David; McGrail, Matthew Richard; Kinsman, Leigh

doi:10.1111/j.1440-1584.2012.01256.x

Cited by 44 publications

(73 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An Australian team of researchers (17) has developed a modified form of the Payback approach, called the Health Services Research Impact Framework (HSRIF). This was designed to better capture the impact of primary health care-specific research, especially in rural contexts.…”

Section: Introductionmentioning

confidence: 99%

Evaluating Research Impact: The Development of a Research for Impact Tool

Tsey

Lawson

Kinchin

et al. 2016

Front. Public Health

View full text Add to dashboard Cite

IntroductionThis paper examines the process of developing a Research for Impact Tool in the contexts of general fiscal constraint, increased competition for funding, perennial concerns about the over-researching of Aboriginal and Torres Strait Islander issues without demonstrable benefits as well as conceptual and methodological difficulties of evaluating research impact. The aim is to highlight the challenges and opportunities involved in evaluating research impact to serve as resource for potential users of the research for impact tool and others interested in assessing the impact of research.Materials and methodsA combination of literature reviews, workshops with researchers, and reflections by project team members and partners using participatory snowball techniques.ResultsAssessing research impact is perceived to be difficult, akin to the so-called “wicked problem,” but not impossible. Heuristic and collaborative approach to research that takes the expectations of research users, research participants and the funders of research offers a pragmatic solution to evaluating research impact. The logic of the proposed Research for Impact Tool is based on the understanding that the value of research is to create evidence and/or products to support smarter decisions so as to improve the human condition. Research is, therefore, of limited value unless the evidence created is used to make smarter decisions for the betterment of society. A practical way of approaching research impact is, therefore, to start with the decisions confronting decision makers whether they are government policymakers, industry, professional practitioners, or households and the extent to which the research supports them to make smarter policy and practice decisions and the knock-on consequences of doing so. Embedded at each step in the impact planning and tracking process is the need for appropriate mix of expertise, capacity enhancement, and collaborative participatory learning-by-doing approaches.DiscussionThe tool was developed in the context of Aboriginal and Torres Strait Islander research but the basic idea that the way to assess research impact is to start upfront with the information needs of decisions makers is equally applicable to research in other settings, both applied (horizontal) and basic (vertical) research. The tool will be further tested and evaluated with researchers over the next 2 years (2016/17). The decision by the Australian Government to include ‘industry engagement’ and ‘impact’ as additions to the Excellence in Research for Australia (ERA) quality measures from 2018 makes the Research for Impact Tool a timely development. The wider challenge is to engage with major Australian research funding agencies to ensure consistent alignment and approaches across research users, communities, and funders in evaluating impact.

show abstract

Section: Introductionmentioning

confidence: 99%

Evaluating Research Impact: The Development of a Research for Impact Tool

Tsey

Lawson

Kinchin

et al. 2016

Front. Public Health

View full text Add to dashboard Cite

show abstract

“…IndicatorsYes/possibleYesYesPossibleYesYesNoPossibleDecision Making Impact ModelLavis, Ross [27]T3, T4Mixed methodsYesYesYesPossibleNo/partialYesPartial/possibleYesEconomic Impact AssessmentDeloitte [41], Glover, Buxton [34]T1–T4Economic assessmentYesNoYesNoYesPartialNoNo/possibleExcellence in Research for AustraliaARC [50]T1–T4Peer-review, Quant. IndicatorsPartialNoPartialNoNoPartialNoNoHealth Services Research Impact FrameworkBuykx, Humphreys [51]T1–T4Quant. Indicators, surveyYesYesYesNo/possiblePartialYes/partialNoPossibleHunter Medical Research Institute Framework to Assess the Impact from Translational Health ResearchSearles [21]T1–T4Mixed MethodsYesYesYesPossibleYesYesYesYesInstitute for Translational Health Sciences Kellogg Logic Model – World Health Organization Health Services Assessment ModelScott, Nagasawa [47]T1–T4Mixed MethodsYesYesYesYes/possiblePartialYesNoPossibleLean/Six-sigma ModelsSchweikhart and Dembe [43]T1–T4Quantitative IndicatorsPartialYesNoPossible/noNoPartialYes/partialYesMatrix Scoring SystemWiegers, Houser [53]T1–T4Quant.…”

Section: Resultsmentioning

confidence: 99%

“…In contrast, some frameworks, such as ARC ERA, EIA and the REF, are designed for retrospective analysis and consequently lend themselves poorly to steering [28, 35, 41, 50]. Finally, a number of the frameworks could accommodate the pre-specification of a health ‘target’ [21, 46, 51, 52]. For example, the Decision Making Impact Model explicitly considers the source of the demand for research knowledge, from which impact assessment through the ‘user-pull’ method could be conducted [27].…”

Section: Resultsmentioning

confidence: 99%

“…The majority of frameworks met this criteria, albeit that the emphasis varies from the relative impact of research outputs [5, 44, 45, 51] to organisational aims, such as progress towards strategic goals [32, 53]. While the methodologies adopted within the frameworks varied, most provide the grounds to facilitate comparison, albeit that greater transparency assists comparison in the “ doing, diffusion and impact of research within and across research fields ” [40].…”

Section: Resultsmentioning

confidence: 99%

“…Inflexible retrospective formats disqualify the existing ARC ERA, EIA, Matrix Scoring System and the Research Engagement for Australia [29, 33, 41, 50, 53]. The Health Services Research Impact Framework and the Research Impact Framework were designed partly to guide researchers in the prospective planning of research implementation [40, 51], but provide limited guidance regarding process metrics that align research to anticipated impacts. MORIA was developed with a consideration to ‘drive desirable behaviours’ [48].…”

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

Measuring research impact in Australia’s medical research institutes: a scoping literature review of the objectives for and an assessment of the capabilities of research impact assessment frameworks

Deeming

Searles

Reeves

et al. 2017

Health Res Policy Sys

View full text Add to dashboard Cite

BackgroundRealising the economic potential of research institutions, including medical research institutes, represents a policy imperative for many Organisation for Economic Co-operation and Development nations. The assessment of research impact has consequently drawn increasing attention. Research impact assessment frameworks (RIAFs) provide a structure to assess research translation, but minimal research has examined whether alternative RIAFs realise the intended policy outcomes. This paper examines the objectives presented for RIAFs in light of economic imperatives to justify ongoing support for health and medical research investment, leverage productivity via commercialisation and outcome–efficiency gains in health systems, and ensure that translation and impact considerations are embedded into the research process.This paper sought to list the stated objectives for RIAFs, to identify existing frameworks and to evaluate whether the identified frameworks possessed the capabilities necessary to address the specified objectives.MethodsA scoping review of the literature to identify objectives specified for RIAFs, inform upon descriptive criteria for each objective and identify existing RIAFs. Criteria were derived for each objective. The capability for the existing RIAFs to realise the alternative objectives was evaluated based upon these criteria.ResultsThe collated objectives for RIAFs included accountability (top-down), transparency/accountability (bottom-up), advocacy, steering, value for money, management/learning and feedback/allocation, prospective orientation, and speed of translation. Of the 25 RIAFs identified, most satisfied objectives such as accountability and advocacy, which are largely sufficient for the first economic imperative to justify research investment. The frameworks primarily designed to optimise the speed of translation or enable the prospective orientation of research possessed qualities most likely to optimise the productive outcomes from research. However, the results show that few frameworks met the criteria for these objectives.ConclusionIt is imperative that the objective(s) for an assessment framework are explicit and that RIAFs are designed to realise these objectives. If the objectives include the capability to pro-actively drive productive research impacts, the potential for prospective orientation and a focus upon the speed of translation merits prioritisation. Frameworks designed to optimise research translation and impact, rather than simply assess impact, offer greater promise to contribute to the economic imperatives compelling their implementation.

show abstract

Measuring health promotion research impact – What researchers think?

Jancey

Burns

Hendriks

et al. 2020

Health Prom J of Aust

View full text Add to dashboard Cite

Issues addressed: The significant investment in health research has resulted in an increased focus to determine suitable approaches to measure health promotion research impact. This research was undertaken at the request of a nationally competitive research organisation that funds health promotion research, to explore how Chief Investigators (CIs) on their funded projects understood research impact, its measurement and associated challenges.Methods: Participants were required to be or have been a CI on an Australian competitive research (exploratory or intervention) grant focussed on health promotion.The qualitative study used thematic analysis from one-on-one interviews. Themes were created and descriptive quotes were selected to illustrate the main findings. Results:The majority of participants were female (n = 13) working at a university (n = 13). Three themes emerged: a) defining health research impact; b) complexity and simplicity of measuring health research impact; c) challenges of measuring health research impact: i) differing language; ii) differing assessment; iii) attribution and timing; and iv) resourcing and skills. Conclusion:Researchers recognised that there are significant challenges in measuring the impact of health promotion research. The most significant factors identified by participants were the disciplinary background of the researchers undertaking the measurement, their skills and experience and the resources (including time) available to assess impact.So what? Research impact assessment is complex, time consuming and requires specific skills to facilitate measurement of impact provision of funding for this activity in research grants and research impact training is required.

show abstract

‘Making evidence count’: A framework to monitor the impact of health services research

Cited by 44 publications

References 15 publications

Evaluating Research Impact: The Development of a Research for Impact Tool

Evaluating Research Impact: The Development of a Research for Impact Tool

Measuring research impact in Australia’s medical research institutes: a scoping literature review of the objectives for and an assessment of the capabilities of research impact assessment frameworks

Measuring health promotion research impact – What researchers think?

Contact Info

Product

Resources

About