Reviewers are blinkered by bibliometrics

Research articles produced through international collaboration are more highly cited than other work, but are they also more novel? Using measures developed by Uzzi et al. (2013), and replicated by Boyack and Klavans (2014), this article tests for novelty and conventionality in international research collaboration. Scholars have found that coauthored articles are more novel and have suggested that diverse groups have a greater chance of producing creative work. As such, we expected to find that international collaboration tends to produce more novel research.Using data from Web of Science and Scopus in 2005, we failed to show that international collaboration tends to produce more novel articles. In fact, international collaboration appears to produce less novel and more conventional knowledge combinations. Transaction costs and communication barriers to international collaboration may suppress novelty. Higher citations to international work may be explained by an audience effect, where more authors from more countries results in greater access to a larger citing community. The findings are consistent with explanations of growth in international collaboration that posit a social dynamic of preferential attachment based upon reputation.

show abstract

“…(Mobley et al, 1992, p. 128). However, they also point out the higher risk of failure (see also Stephan et al, (2017)).…”

Section: -Atypical Combinations Reflecting Novel Contributionsmentioning

confidence: 98%

International research collaboration: Novelty, conventionality, and atypicality in knowledge recombination

2019

View full text Add to dashboard Cite

show abstract

“…conceptualized interdisciplinarity in terms of both diversity and coherence. Analogously, others use the words "novelty" and "conventionality" (Uzzi et al, 2013;Schilling & Green, 2015;Stephan et al, 2017), or 'atypical' combinations, but which are limited in terms of accounting for balance. Leydesdorff & Rafols (2011, at p. 852) and Rafols, Leydesdorff, O'Hare, Nightingale, & Stirling (2012, at p. 1268) have proposed to operationalize coherence as follows:…”

Section: Disparity or Similaritymentioning

confidence: 99%

Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient

Leydesdorff

Wagner

Bornmann

2019

Journal of Informetrics

127

View full text Add to dashboard Cite

Questions of definition and measurement continue to constrain a consensus on the measurement of interdisciplinarity. Using Rao-Stirling (RS) Diversity produces sometimes anomalous results.We argue that these unexpected outcomes can be related to the use of "dual-concept diversity" which combines "variety" and "balance" in the definitions (ex ante). We propose to modify RS Diversity into a new indicator (DIV) which operationalizes "variety," "balance," and "disparity" independently and then combines them ex post. "Balance" can be measured using the Gini coefficient. We apply DIV to the aggregated citation patterns of 11,487 journals covered by the Journal Citation Reports 2016 of the Science Citation Index and the Social Sciences CitationIndex as an empirical domain and, in more detail, to the citation patterns of 85 journals assigned to the Web-of-Science category "information science & library science" in both the cited and citing directions. We compare the results of the indicators and show that DIV provides improved results in terms of distinguishing between interdisciplinary knowledge integration (citing) versus knowledge diffusion (cited). The new diversity indicator and RS diversity measure different features. A routine for the measurement of the various operationalizations of diversity (in any data matrix) is made available online.

show abstract

“…To make things worse, criteria for evaluation are not universal either, and even if they were, they are plagued with vague, difficult to objectivize terms such as "scientific quality", "innovation" or "creativity" [47,48]. Contrary to the scientific practice itself, scientific evaluation methods are based on notoriously vague concepts and therefore lack impartiality, understanding impartiality as "the ability for any observer to recognize evidence as evidence and to see the bearing of evidence on theory in the same way" impact [24]. Our results suggest that these simple measurements of "scientific quality" have very limited predictive value of future performance once the top candidates have been preselected.…”

Section: Methods and Results For Details)mentioning

confidence: 99%

Talent Identification at the limits of Peer Review: an analysis of the EMBO Postdoctoral Fellowships Selection Process

Klaus¹,

Alamo

2018

Preprint

View full text Add to dashboard Cite

Scientific peer review is still the most common system for fund allocation despite having been shown in multiple instances to lack accuracy in identifying the most meritorious applications among high quality ones.This study evaluates two aspects of the selection process of the topranked applicants to the EMBO Long-Term Fellowship program in 2007.First, the accuracy of the system is evaluated by comparing the level of career progression of the candidates in 2017 with the original award decisions made in 2007. The second aspect, explores the relationship of career progression with indicators derived from the information available to evaluators at the time of application. The results obtained suggest that the peer review system is not substantially better than random selection in identifying the best candidates once an initial pre-selection of the most promising ones is performed. Not only that, the analysis of the indicators studied, some of which have not been analyzed in detail in the past, suggests that among other potential sources of uncertainty, the information available at the time of application is not sufficiently predictive of career progression. As previously described, however, we find differences in career progression between men and women. We propose a new mixed model of fellowship evaluation in which peer review is used to select high quality applications, and random allocation of funds is subsequently used to award fellowships among these top ranked candidates.

show abstract

Reviewers are blinkered by bibliometrics

Cited by 153 publications

References 10 publications

International research collaboration: Novelty, conventionality, and atypicality in knowledge recombination

International research collaboration: Novelty, conventionality, and atypicality in knowledge recombination

Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient

Talent Identification at the limits of Peer Review: an analysis of the EMBO Postdoctoral Fellowships Selection Process

Contact Info

Product

Resources

About