Systematic analysis of agreement between metrics and peer review in the UK REF

Abstract

When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found. However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review. We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review. In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF. The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review. To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree quite well with peer review and may offer an alternative to peer review.

Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation

“…Leaving potential caveats aside (see also footnote 14), the studies by Auspurg et al (2015), Diekmann et al (2012), Pride and Knoth (2018), Traag and Waltman (2019), and Harzing (2017) could be interpreted as seeming to exhibit-similarly as in other areas of research on heuristics-a 'less-is-equal effect': a complex procedure uses more information than a simple tool and performs extensive assessments, but nevertheless lead to similar final judgments. Precisely specifying when, how, and why such effects emerge could be the goal of a foundational research program on bibliometrics.…”

Evaluating research and researchers by the journal impact factor: Is it better than coin flipping?

“…However, this is not strictly true because scientific relevance and number of citations are only correlated. This notion is behind a large number of publications (e.g., De Bellis, 2009) but it is difficult to study because it requires a numerical scale for the scientific relevance; the best support of this correlation is found in studies that compare citation counts and peer review assessments (Traag & Waltman, 2018;Wilsdon et al, 2015).…”

Scientometrics and management of scientific activities: once again about the global and Ukrainian

“…Тому активно розробляються, обговорюються та використовуються й інші методи, зокрема кількісний аналіз бібліометричних та інших релевантних до науки даних. Уже є чимало робіт, присвячених дослідженню кореляцій між експертними оцінками та кількісними індикаторами (напр., [4,5]), проте одностайності досягнуто наразі лише у висновку, що наукометрики не можуть розглядатися як заміна експертній оцінці, а є лише комплементарними до неї.…”