Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool

Abstract

BackgroundThere is a growing emphasis on the importance of research having demonstrable public benefit. Measurements of the impacts of research are therefore needed. We applied a modified impact assessment process that builds on best practice to 5 years (2003–2007) of intervention research funded by Australia’s National Health and Medical Research Council to determine if these studies had post-research real-world policy and practice impacts.MethodsWe used a mixed method sequential methodology whereby chief investigators of eligible intervention studies who completed two surveys and an interview were included in our final sample (n = 50), on which we conducted post-research impact assessments. Data from the surveys and interviews were triangulated with additional information obtained from documentary analysis to develop comprehensive case studies. These case studies were then summarized and the reported impacts were scored by an expert panel using criteria for four impact dimensions: corroboration; attribution, reach, and importance.ResultsNineteen (38%) of the cases in our final sample were found to have had policy and practice impacts, with an even distribution of high, medium, and low impact scores. While the tool facilitated a rigorous and explicit criterion-based assessment of post-research impacts, it was not always possible to obtain evidence using documentary analysis to corroborate the impacts reported in chief investigator interviews.ConclusionsWhile policy and practice is ideally informed by reviews of evidence, some intervention research can and does have real world impacts that can be attributed to single studies. We recommend impact assessments apply explicit criteria to consider the corroboration, attribution, reach, and importance of reported impacts on policy and practice. Impact assessments should also allow sufficient time between impact data collection and completion of the original research and include mechanisms to obtain end-user input to corroborate claims and reduce biases that result from seeking information from researchers only.

Therapy interventions for children with neurodisabilities: a qualitative scoping study

“…This very much concurs with the wider accepted view of the current situation 1,6,59 and therapists' views on the barriers to evidence-based practice. [60][61][62][63][64][65][66][67][68] These findings are not unique to the therapy professions investigated in this study, and have been reported across a wide range of health-care professions. [64][65][66] Furthermore, and as other studies report, [61][62][63] the applicability of existing, higher-quality evidence (derived from studies not conducted in routine practice or clinical settings) was questioned.…”

“…First, Morris et al's study was restricted to 'interventions and process': this was not specifically defined and might not have included the notion that service-/organisational-level issues and characteristics can be understood as interventions in themselves. 68 Second, this study would not have included the identification of methodological or more 'foundational' research within its scope. Third, the research prioritisation approach by Morris et al…”

Knowledge management for policy impact: the case of the European Commission’s Joint Research Centre

“…Second, it is easier to monitor and evaluate the impact of knowledge on inputs and outputs than outcomes (Cohen et al 2015), which can produce an over-reliance on short-term impacts. It can encourage attempts to 'game' systems to produce too-heroic stories of the impact of key individuals (Dunlop, 2018).…”

Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?

“…Societal impact measurements are mostly commissioned by governments which argue that measuring the impact on science says little about real-world benefits of research (Cohen et al 2015). Nightingale and Scott (2007) summarize this argumentation in the following pointedly sentence: “Research that is highly cited or published in top journals may be good for the academic discipline but not for society” (p. 547).…”

“…Others however see case studies as ‘fairy tales of influence’ and argue for a more consistent toolkit of impact metrics that can be more easily compared across and between cases” (p. 49, see also Atkinson 2014). For Cohen et al (2015) “the holy grail is to find short term indicators that can be measured before, during or immediately after the research is completed and that are robust predictors of the longer term impact … from the research”. Ovseiko et al (2012) think that important areas of impact can only be captured qualitatively.…”