Evaluating Journal Impact Factor: a systematic survey of the pros and cons, and overview of alternative measures

Abstract

Background: Journal Impact Factor (JIF) has several intrinsic flaws, which highlight its inability to adequately measure citation distributions or indicate journal quality. Despite these flaws, JIF is still widely used within the academic community, resulting in the propagation of potentially misleading information. A critical review of the usefulness of JIF is needed including an overview of the literature to identify viable alternative metrics. The objectives of this study are: (1) to assess the usefulness of JIF by compiling and comparing its advantages and disadvantages; (2) to record the differential uses of JIF within research environments; and (3) to summarize and compare viable alternative measures to JIF. Methods: Three separate literature search strategies using MEDLINE and Web of Science were completed to address the three study objectives. Each search was completed in accordance with PRISMA guidelines. Results were compiled in tabular format and analyzed based on reporting frequency. Results: For objective (1), 84 studies were included in qualitative analysis. It was found that the recorded advantages of JIF were outweighed by disadvantages (18 disadvantages vs. 9 advantages). For objective (2), 653 records were included in a qualitative analysis. JIF was found to be most commonly used in journal ranking (n = 653, 100%) and calculation of scientific research productivity (n = 367, 56.2%). For objective (3), 65 works were included in qualitative analysis. These articles revealed 45 alternatives, which includes 18 alternatives that improve on highly reported disadvantages of JIF. Purpose Assessing the usefulness of JIF Assessing differential uses of JIF Overview of alternative measures

Evaluating the quality of systematic reviews of comparative studies in autograft‐based anterior cruciate ligament reconstruction using the AMSTAR‐2 tool: A systematic umbrella review

“…Conversely, the impact factor, in particular, was found to have a significant association with AMSTAR assessments in other orthopaedic fields (e.g., THA/TKA) [71]. It is unclear why there lies this discrepancy, though there is currently discourse in the scientific community disputing the true validity of journal impact factor and its implications for evaluating studies and deriving assumptions regarding their overall quality from such statistics [43]. The number of citations not having an association with AMSTAR‐2 ratings poses an important distinction to be made when interpreting citation numbers—this metric may not correlate with true study quality and should be used with caution as a means of critical appraisal.…”

Biomarker adoption in developmental science: A data‐driven modelling of trends from 90 biomarkers across 20 years

“…Our use of one impact factor score, SJR, in our secondary analyses to assess the influence of journal reach similar limits the interpretation of our results. There are over 45 different metrics for journal impact, and while some, such as Journal Impact Factor and SJR, are strongly correlated, each metric provides a slightly different measure of journal reach and influence (Glänzel & Moed, 2002; Mech et al, 2020; Saha, 2003). We used SJR because it was both a prominent method of assessing impact and easily merged with our 430,000 articles.…”

Novel Journal Metrics in Cardiothoracic Surgery: How Different Are Contemporary Metrics From the Impact Factor?

“…These include its lack of comparability across journals of different specialties as well as its susceptibility to being positively skewed by a few articles with higher citation counts. 1 Journals striving to achieve or maintain a high IF may be tempted to publish articles deemed to have greater citation potential, such as review articles. 2 These limitations, and the growing influence of information dissemination via social media, have driven the creation of newer journal metrics, such as the h-index, g-index, and Altmetric Attention Score (AAS).…”