Assessing Health Research and Innovation Impact: Evolution of a Framework and Tools in Alberta, Canada

Abstract

Publicly funded research and innovation (R&I) organizations around the world are facing increasing demands to demonstrate the impacts of their investments. In most cases, these demands are shifting from academically based outputs to impacts that benefit society. Funders and other organizations are grappling to understand and demonstrate how their investments and activities are achieving impact. This is compounded with challenges that are inherent to impact assessment, such as having an agreed understanding of impact, the time lag from research to impact, establishing attribution and contribution, and consideration of diverse stakeholder needs and values. In response, many organizations are implementing frameworks and using web-based tools to track and assess academic and societal impact. This conceptual analysis begins with an overview of international research impact frameworks and emerging tools that are used by an increasing number of public R&I funders to demonstrate the value of their investments. From concept to real-world, this paper illustrates how one organization, Alberta Innovates, used the Canadian Academy of Health Sciences (CAHS) impact framework to guide implementation of its fit-for-purpose impact framework with an agnostic international six-block protocol. The implementation of the impact framework at Alberta Innovates is also supported by adopting emerging web-based tools. Drawing on the lessons learned from this continuous organizational endeavor to assess and measure R&I impact, we present preliminary plans for developing an impact strategy for Alberta Innovates that can be applied across sectors, including energy, environment and agriculture, and may possibly be adopted by other international funders.

The impact of conducting preclinical systematic reviews on researchers and their research: A mixed method case study

“…Consequently, these evaluations usually assess all directions and categories of impacts in a structured and high-quality manner [ 22 ]. This may include the use of complex, time-intensive frameworks, including impact pathways to identify or foresee how research projects create (expected) impacts, for example the societal or environmental impact of a research project [ 23 , 24 ]. Since impact evaluations began to be conducted by universities in the 1970’s and 1980’s, a plethora of frameworks has been proposed, with different scopes, aims, and assets [ 23 – 25 ].…”

“…Each proposes a specific approach to assessing research knowledge and research quality, and to measuring impacts [ 26 ]. Well-known frameworks include the Canadian Academy of Health Sciences Preferred Framework, the National Institute of Health Research Dashboard, or the Excellence in Research for Australia [ 24 , 27 , 28 ]. Such large endeavours were considered out of scope for our current project.…”

The Health Innovation Impact Checklist: a tool to improve the development and reporting of impact models for global health innovations

“…Donor funding and investments from multilateral, bilateral, and global health initiatives are increasingly relying on performance-based measures [ 2–5 ]. In addition, donor demands are shifting from technical and academic outputs to impacts that measurably benefit society [ 6 ]. To demonstrate efficiency and garner donor and private sector interest, global health implementing agencies need to be able to measure and report the impact of their interventions on health outcomes.…”

Research Metrics for Health Science Schools: A Conceptual Exploration and Proposal

“…However, despite some existing models, there is no universally accepted set of metrics to assess ROI for public health research; despite a growing demand to identify and develop standards to effectively measure and communicate research-related progress and success (Banzi et al, 2011 ). In addition, challenges such as inconsistent definitions of outcomes, time lapse between research and impact, issues in establishing attribution, and accommodating stakeholder perceptions make this goal complex (Graham et al, 2018 ). A universally accepted approach however, would provide a system to measure social, economic, and environmental benefits of research.…”