Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?

Abstract

In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA), societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric—a start-up providing publication level metrics—started to make data for publications available which have been mentioned in policy documents. We regard this data source as an interesting possibility to specifically measure the (societal) impact of research. Using a comprehensive dataset with publications on climate change as an example, we study the usefulness of the new data source for impact measurement. Only 1.2 % (n = 2341) out of 191,276 publications on climate change in the dataset have at least one policy mention. We further reveal that papers published in Nature and Science as well as from the areas “Earth and related environmental sciences” and “Social and economic geography” are especially relevant in the policy context. Given the low coverage of the climate change literature in policy documents, this study can be only a first attempt to study this new source of altmetrics data. Further empirical studies are necessary, because mentions in policy documents are of special interest in the use of altmetrics data for measuring target-oriented the broader impact of research.

What Can Altmetric.com Tell Us About Policy Citations of Research? An Analysis of Altmetric.com Data for Research Articles from the University of Sheffield

“…This means that 0.65% of Sheffield research, across all disciplines, is cited by at least one policy document. This compares relatively well with the overall impact of research on policy according to previous research based on data from papers indexed in WoS, which was just 0.5% from (n = 11,254,636) (Haunschild and Bornmann, 2017), and 1.2% of published climate change research from (n = 191,276) (Bornmann et al, 2016). …”

“…Recently the question has been asked whether altmetrics can measure research impact on policy (Bornmann et al, 2016) (Haunschild and Bornmann, 2017). Waltman and Costas (2014) reflected: "altmetrics opens the door to a broader interpretation of the concept of impact and to more diverse forms of impact analysis."…”

“…It is perhaps not surprising that the majority of the research with greatest policy impact had been undertaken within the fields of economics and health (see Tables 1 and 4): alongside education, these are arguably the two major areas of policy development nationally and internationally (Haunschild and Bornmann, 2017), with the environment being another focus of policy work (Bornmann et al, 2016). It is, therefore, no surprise to see the relative numbers of research articles, categorized by Faculty, with medicine and health far outstripping any other discipline.…”

The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future

“…Rather, it is the fact that, through different meanings, its products are pushed outside of the academic arena in an attempt to reach society. This way of interpreting the impact of research has opened a very prolific avenue of research focused on so-called altmetrics, i.e., alternative metrics (Barbaro, Gentili, & Rebuffi, 2014;Barnes, 2015;Bornmann, 2015aBornmann, , 2015bBornmann, Haunschild, & Marx, 2016;Hammarfelt, 2014). According to Shema, Bar-Ilan and Thelwall (2014, p. 19), they refer to: "web-based metrics for the impact of scholarly material, with an emphasis on social media outlets as sources of data".…”

Advancing the management of obstructive airways diseases through translational research

“…The final challenge, particularly relevant to basic science, is timing . Research impacts may take decades to materialize …”