Purpose
Pay, tenure and promotion decisions are frequently based upon inferences regarding the value of faculty research. Meanwhile, departmental, college and university reputations are frequently based on perceptions regarding the quality of research being produced by its faculty. Making correct inferences requires accurate measurement of research quality, which is often based upon the journal through which results are shared. This research expands upon the research found elsewhere through its detailed investigation of leading journals in two business disciplines, including examination of four different citation-based measures and four journal characteristics which are exogenous to the quality of any individual piece of research. The paper aims to discuss this issue.
Design/methodology/approach
This study assists in the development of an accurate perspective regarding research quality, by studying the popular Journal Citation Reports (JCR) impact factor. A further expansion on the past literature is consideration of three newer journal quality metrics: SCImago Journal Rank (SJR), Source Normalized Impact per Paper (SNIP) and percentage of articles cited. Top-tier journals in finance and information systems are compared to evaluate the consistency of these measures across disciplines. Differences in journal characteristics and their impact on citation-rate based measures of quality are also examined. The potential impact of discipline-based variation in acceptance rate, issue frequency, the time since journal inception and total reviewers are put forth as additional potential exogenous factors that may impact the perception of journal quality. t-Tests are employed for discipline comparisons, while correlation and multiple regression are used for journal characteristic analysis.
Findings
There is a significant difference in the JCR impact measures of high-quality finance journals vs high-quality information systems journals, which are correlated with a variety of journal-specific factors including the journal’s acceptance rate and frequency of issue. Information systems journals domination of finance journals persists whether one considers mean, median, minimum or maximum impact factors. SJR measures for finance journals are consistently higher than information systems journals, though the SJR value of any individual journal can be quite volatile. By comparison, the SNIP metric rates premier information systems journals higher. Over 12 percent more of the articles in leading information systems journals are cited during the initial three years.
Research limitations/implications
Logical extensions of this research include examining journals in other business disciplines. One could also evaluate quality measures reaction to variation in journal characteristics (i.e. changes in acceptance rates). Furthermore, one could include other measures of journal quality, including the recently released CiteScore metric. Such research will build on the present research and improve the accuracy of research quality assessment.
Practical implications
To the extent that citation-based research measures and journal-specific factors vary across disciplines as demonstrated by our investigation, discipline-specific traits should be considered adjusted for, when making inferences about the long-term value of recently published research. For instance, finance faculty publishing in journals with JCR readings of 2.0 are in journals that are 53 percent above the discipline’s average, while information systems faculty publishing in journals with JCR readings of 2.0 are in journals that are 18 percent below the discipline’s average. Furthermore, discipline-specific differences in journal characteristics, leading to differences in citation-based quality measures, should be considered when making inferences about the long-term value of recently published research in the process of making recommendations regarding salary adjustments, retention and promotion.
Social implications
Quantity and quality of research are two hallmarks of leading research institutions. Assessing research quality is very problematic because its definition has changed from being based on review process (i.e. blind refereed), to acceptance rates, to impact factors. Furthermore, the impact factor construct has been a lightning rod of controversy as researchers, administrators and journals themselves argue over which metric to employ. This research is attempting to assess how impact factors and journal characteristics may influence the impact factors, and how these interactions vary business discipline. The research is especially important and relevant to the authors which separately chair departments including finance and information systems faculty, and therefore are in roles requiring assessment of faculty research productivity including quality.
Originality/value
This study is a detailed analysis of bibliographic aspects of the top-tier journals in two quantitative business areas. In addition to the popular JCR, SJR and SNIP measures of performance, the analysis studies the seldom-examined percentage of the article cited metric. A deeper understanding of citation-based measures is obtained though the evaluation of changes in how journals have been rated on these metrics over time. The research shows that there are discipline-related systematic differences in both citation-based research measures and journal-specific factors and that these discipline-specific traits should be considered when making inferences about the long-term value of recently published research. Furthermore, discipline-specific difference in journal characteristics, leading to differences in citation-based quality measures, should be considered when making personnel and remuneration decisions.
