Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews

Abstract

The Journal Impact Factor is often used as a proxy measure for journal quality, but the empirical evidence is scarce. In particular, it is unclear how peer review characteristics for a journal relate to its impact factor. We analysed 10,000 peer review reports submitted to 1,644 biomedical journals with impact factors ranging from 0.21 to 74.7. Two researchers hand-coded sentences using categories of content related to the thoroughness of the review (Materials and Methods, Presentation and Reporting, Results and Discussion, Importance and Relevance) and helpfulness (Suggestion and Solution, Examples, Praise, Criticism). We fine-tuned and validated transformer machine learning language models to classify sentences. We then examined the association between the number and percentage of sentences addressing different content categories and 10 groups defined by the Journal Impact Factor. The median length of reviews increased with higher impact factor, from 185 words (group 1) to 387 words (group 10). The percentage of sentences addressing Materials and Methods was greater in the highest Journal Impact Factor journals than in the lowest Journal Impact Factor group. The results for Presentation and Reporting went in the opposite direction, with the highest Journal Impact Factor journals giving less emphasis to such content. For helpfulness, reviews for higher impact factor journals devoted relatively less attention to Suggestion and Solution than lower impact factor journals. In conclusion, peer review in journals with higher impact factors tends to be more thorough, particularly in addressing study methods while giving relatively less emphasis to presentation or suggesting solutions. Differences were modest and variability high, indicating that the Journal Impact Factor is a bad predictor of the quality of peer review of an individual manuscript.

Prevalence of Short Peer Reviews in 3 Leading General Medical Journals

“…These results were consistent across different word count thresholds used to define very short reviews. Review word count has been shown to be positively associated with journal impact factor; thus, these findings may be associated with differences in journal impact factors (journal impact factor: 96.21 for The BMJ vs 11.61 for PLOS Medicine vs 11.81 for BMC Medicine ) . However, the mechanisms underlying this association are not fully clear.…”

Prisoner's science

“…Given this difficult situation, we can only advise scientists to resist the temptation of judging work based on the prestige of a journal and to take solace in the fact that good science rises to the top no matter where it is published. Journal impact factor is in fact a poor predictor of the quality of peer review or the citation rate of individual papers ( 33 , 34 ), and high impact journals have some of the highest retraction rates ( 35 ). Some of the most important scientific papers in history were published in venues that today may be less well known to scientists in the biomedical sciences, such as Griffith's discovery of the transforming principle ( Journal of Hygiene , reference 36 ), Krebs' discovery of the tricarboxylic acid cycle ( Enzymologia , reference 37 ), Glick's demonstration that the bursa of Fabricius is required for the production of antibodies by B cells in newly hatched chicks ( Poultry Science , reference 38 ), and Shimomura's discovery of jellyfish associated fluorescence ( Journal of Cellular and Comparative Physiology , reference 39 ), which ultimately led to the discovery of green fluorescent protein in jellyfish ( 40 ).…”

The Responsible Research(er) Recruitment Checklist: A best practice guide for applying principles of responsible research assessment in researcher recruitment materials

“…One possible explanation is that such journals prioritise novel findings and neat narratives over methodological rigour. The JIF is also a poor predictor of peer review quality (Severin et al, 2023), again suggesting that a high JIF is not necessarily indicative of high research quality (and perhaps the opposite). For a more complete overview of JIFs and their limitations, see Larivière and Sugimoto (2019).…”