Scientific journals still matter in the era of academic search engines and preprint archives

Abstract

Journals play a critical role in the scientific process because they evaluate the quality of incoming papers and offer an organizing filter for search. However, the role of journals has been called into question because new preprint archives and academic search engines make it easier to find articles independent of the journals that publish them. Research on this issue is complicated by the deeply confounded relationship between article quality and journal reputation. We present an innovative proxy for individual article quality that is divorced from the journal's reputation or impact factor: the number of citations to preprints posted on http://arxiv.org. Using this measure to study three subfields of physics that were early adopters of arXiv, we show that prior estimates of the effect of journal reputation on an individual article's impact (measured by citations) are likely inflated. While we find that higher‐quality preprints in these subfields are now less likely to be published in journals compared to prior years, we find little systematic evidence that the role of journal reputation on article performance has declined.

Misinformation in and about science

“…For producers, the story changes somewhat. We find that papers highly cited as preprints are less likely to be published in journals at all (98).…”

“…In a recent study, we tracked citations of papers published on the arXiv before and after journal publication (98). After controlling for article quality, we find that arXiv articles published in higher-ranked journals received more citations than articles published in lower-tier journals.…”

Use of the journal impact factor for assessing individual articles: Statistically flawed or not?

“…In a dynamic model with multiple time periods, the IF of a journal can be calculated in a more realistic way (by using appropriate publication and citation windows) and may evolve over time. Moreover, in a dynamic model, the citations of the articles published in a journal may not only determine the IF of the journal but may also be influenced by the IF of the journal in earlier time periods, creating a kind of Matthew effect of the IF (for further discussion on this possibility, see Kim et al, 2020;Larivière & Gingras, 2010;Traag, in press). A more advanced simulation model may also consider that the peer review carried out by journals takes time and that researchers may not want to risk delaying publication of their work by submitting it to a journal by which it will most likely be rejected.…”

“…One of the most common concerns relates to the use of the IF for assessing individual articles. It is often argued that from a statistical point of view it is incorrect, or at least highly problematic, to use journal-level indicators such as the IF in the assessment of individual articles (e.g., Garfield, 2006 ; Gingras, 2016 ; Larivière et al, 2016 ; Leydesdorff et al, 2016 ; Seglen, 1992 ; Seglen, 1997 ; Zhang et al, 2017 ). This point is also made in the well-known San Francisco Declaration on Research Assessment ( DORA, 2013 ).…”

“…A high-profile paper by Larivière et al (2016) again draws attention to the skewness of the distribution of citations over the articles in a journal. The authors recommend that, when a journal publishes its IF, it should also publish the underlying citation distribution.…”