Interrater reliability and convergent validity of <scp>F</scp>1000<scp>P</scp>rime peer review

Bornmann, Lutz

doi:10.1002/asi.23334

Cited by 31 publications

(24 citation statements)

References 57 publications

(92 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The correlations between the F1000 score and the bibliometric indicators are on a significantly lower level. However, the lower coefficients are in agreement with the results of other studies which correlated F1000 scores and citation metrics: The meta‐analysis of Bornmann () reveals a pooled r = 0.25. The pooled coefficient and the coefficients in Table can be interpreted as between low and medium, whereas CP correlates the lowest ( r s = 0.23) and SNCS (2) the highest ( r s = 0.31) with the F1000 scores.…”

Section: Resultssupporting

confidence: 89%

Relative Citation Ratio (RCR): An empirical attempt to study a new field‐normalized bibliometric indicator

Bornmann

Haunschild

2016

Asso for Info Science & Tech

Self Cite

View full text Add to dashboard Cite

Hutchins, Yuan, Anderson, and Santangelo (2015) proposed the Relative Citation Ratio (RCR) as a new field‐normalized impact indicator. This study investigates the RCR by correlating it on the level of single publications with established field‐normalized indicators and assessments of the publications by peers. We find that the RCR correlates highly with established field‐normalized indicators, but the correlation between RCR and peer assessments is only low to medium.

show abstract

Section: Resultssupporting

confidence: 89%

Relative Citation Ratio (RCR): An empirical attempt to study a new field‐normalized bibliometric indicator

Bornmann

Haunschild

2016

Asso for Info Science & Tech

Self Cite

View full text Add to dashboard Cite

show abstract

“…Bornmann and Leydesdorff (), Li and Thelwall (), Mohammadi and Thelwall (), and Waltman and Costas () calculated coefficients for the correlation between F1000Prime scores and citation impact. The meta‐analysis of Bornmann () including the coefficients of these studies revealed a pooled r = .25, which can be interpreted as a medium effect size. The result of the meta‐analysis with a relatively low pooled r might be a sign that F1000Prime scores are not able to reflect the concept of quality properly and are not representative of “similar” journal peer‐review processes.…”

Section: Discussionmentioning

confidence: 95%

“…The result of the meta‐analysis with a relatively low pooled r might be a sign that F1000Prime scores are not able to reflect the concept of quality properly and are not representative of “similar” journal peer‐review processes. Although this study is based on manuscripts with at least two scores by different faculty members (to increase the reliability of the scores), the low interreviewer reliability revealed by Bornmann () also questions the link between quality and F1000Prime scores.…”

Section: Discussionmentioning

confidence: 99%

“…The author of this study has already used the extensive F1000Prime data set for the investigation of different research topics: Bornmann () investigates the reliability and predictive validity of the experts' ratings in the F1000Prime peer review system. Bornmann (, ) used the data to investigate whether alternative metrics (altmetrics, see Bornmann, ) are able to measure the societal impact of papers.…”

Section: Literature Overview and Purpose Of The Studymentioning

confidence: 99%

See 1 more Smart Citation

Is collaboration among scientists related to the citation impact of papers because their quality increases with collaboration? An analysis based on data from F1000Prime and normalized citation scores

Bornmann

2016

Asso for Info Science & Tech

Self Cite

View full text Add to dashboard Cite

In recent years, the relationship of collaboration among scientists and the citation impact of papers have been frequently investigated. Most of the studies show that the two variables are closely related: An increasing collaboration activity (measured in terms of number of authors, number of affiliations, and number of countries) is associated with an increased citation impact. However, it is not clear whether the increased citation impact is based on the higher quality of papers that profit from more than one scientist giving expert input or other (citation‐specific) factors. Thus, the current study addresses this question by using two comprehensive data sets with publications (in the biomedical area) including quality assessments by experts (F1000Prime member scores) and citation data for the publications. The study is based on more than 15,000 papers. Robust regression models are used to investigate the relationship between number of authors, number of affiliations, and number of countries, respectively, and citation impact—controlling for the papers' quality (measured by F1000Prime expert ratings). The results point out that the effect of collaboration activities on impact is largely independent of the papers' quality. The citation advantage is apparently not quality related; citation‐specific factors (e.g., self‐citations) seem to be important here.

show abstract

“…Nonetheless, recent literature has found a correlation between expert opinion and citation impact. 14 Furthermore, assessments of return on investment should also consider other factors, such as the type of research. We recently reported that at NIMH, basic science projects appear to yield a greater return on investment than applied projects 6 ; we are planning similar analyses at NHLBI.…”

mentioning

confidence: 99%

Predicting Productivity Returns on Investment

Lauer

Danthi

Kaltman

et al. 2015

Circulation Research

View full text Add to dashboard Cite

There are conflicting data regarding the ability of peer review percentile rankings to predict grant productivity, as measured through publications and citations. To understand the nature of these apparent conflicting findings, we analyzed bibliometric outcomes of 6873 de novo cardiovascular R01 grants funded by the National Heart, Lung, and Blood Institute between 1980 and 2011. Our outcomes focus on “Top-10%” papers, meaning papers that were cited more often than 90% of other papers on the same topic, of the same type (e.g. article, editorial), and published in the same year. The 6873 grants yielded 62,468 papers, of which 13,507 (or 22%) were Top-10% papers. There was a modest association between better grant percentile ranking and number of top-10% papers. However, discrimination was poor (area under ROC 0.52, 95% CI 0.51–0.53). Furthermore, better percentile ranking was also associated with higher annual and total inflation-adjusted grant budgets. There was no association between grant percentile ranking and grant outcome as assessed by number of top-10% papers per $million spent. Hence, the seemingly conflicting findings regarding peer review percentile ranking of grants and subsequent productivity largely reflect differing questions and outcomes. Taken together, these findings raise questions about how best NIH should use peer review assessments to make complex funding decisions.

show abstract

Interrater reliability and convergent validity of F1000Prime peer review

Cited by 31 publications

References 57 publications

Relative Citation Ratio (RCR): An empirical attempt to study a new field‐normalized bibliometric indicator

Relative Citation Ratio (RCR): An empirical attempt to study a new field‐normalized bibliometric indicator

Is collaboration among scientists related to the citation impact of papers because their quality increases with collaboration? An analysis based on data from F1000Prime and normalized citation scores

Predicting Productivity Returns on Investment

Contact Info

Product

Resources

About