Citations as a Measure of the Research Outputs of New Zealand’s Economics Departments: The Problem of ‘Long and Variable Lags’

2013

Research Evaluation

New Zealand's academic research assessment scheme, the Performance Based Research Fund (PBRF), was launched in 2002 with the stated objective of increasing research quality in the nation's universities. Evaluation rounds were conducted in 2003, 2006 and 2012. In this paper, we employ 22 different journal weighting schemes to generate output estimates of refereed journal paper and page production over three six year periods (1994-1999; 2000-2005 and 2006-2011). These time periods reflect a pre-PBRF environment, a mixed assessment period, and a pure PBRF research environment, respectively. Our findings indicate that, on average, research productivity, defined in either paper or page terms, has increased since the introduction of the PBRF. However, this outcome is due to a major increase in the quantity of papers and pages produced per capita that has more than offset a decline in the quality of published outputs since the introduction of the PBRF. In other words, our findings suggest that the PBRF has failed to achieve its stated goal of increasing average research quality, but it has resulted in substantial gains in productivity achieved via large increases in the quantity of refereed journal articles.

mentioning

confidence: 99%

The New Zealand performance-based research fund and its impact on publication activity in economics

2013

Research Evaluation

“…The "narrow" approach is to grant recognition only to journals listed under the economics discipline, and to ignore the many journals in "border" areas such as finance, urban and regional studies and policy. 31 Although this figure is distressing low, it must be stressed that approximately 44 percent of the refereed journal articles generated by New Zealand economists, albeit over a slightly different time period (2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008), were published in journals not included in the ISI database (see Tressler and Anderson, 2012). It should also be noted that if we add back self-cites, the estimate increases to 29.2 percent.…”

Section: Resultsmentioning

confidence: 99%

“…However, the suitability of a two year citation attribution period is questionable for economics, and, more generally, the social sciences. Given the normal time period required for manuscript preparation, refereeing, revision, and publication, a two year cut-off for citations seems extremely tight (Tressler and Anderson, 2012). The two year lag structure was developed in the context of the biological sciences, a discipline with publication practices and lags much different than those prevailing in economics.…”

Section: Basic Citation-based Impact Factorsmentioning

confidence: 99%

The Impact of Journal Weighting Scheme Characteristics on Research Output Measurement in Economics: The New Zealand Case

Review of Economics and Institutions

2012

Self Cite

Abstract: In this study we test for the ‘power’ or aggressiveness of various journal weighting schemes, especially those based on the recursive adjustment methodology first developed by Liebowitz and Palmer. Using data generated by New Zealand’s academic economists, we provide quantitative measures of the differences between recursive adjustment-based schemes and selected alternatives. We then compare the performance of economics departments under each of our journal weighting schemes and, for comparison purposes, one based on direct citation counts. We find departmental rankings based on selected recursive adjustment schemes to be relatively stable, but these rankings differed substantially from those generated by our alternative schemes. This suggests that departmental hiring practices and research strategies must be sensitive to the type of funding scheme employed. In particular, research on domestic and regional issues is likely to be unattractive to researchers if a high-powered journal weighting scheme is adopted as the “official” standard since regional journals, the natural outlet for such work, are frequently zero-weighted by such schemes.

“…For example, see Nederhof & van Raan, 1993;Adams, 2005;Van Leeuwen, 2006;Levitt & Thelwall, 2008, 2011Anderson & Tressler, 2013;Tressler & Anderson, 2012;Bertocchi et al, 2013;Wang, 2013;Stern, 2014;and Burns & Stern, 2015. In general, the findings provide some support for using direct citation counts to papers in the sciences, but little support for doing so in the social sciences. For example, see Nederhof & van Raan, 1993;Adams, 2005;Van Leeuwen, 2006;Levitt & Thelwall, 2008, 2011Anderson & Tressler, 2013;Tressler & Anderson, 2012;Bertocchi et al, 2013;Wang, 2013;Stern, 2014;and Burns & Stern, 2015. In general, the findings provide some support for using direct citation counts to papers in the sciences, but little support for doing so in the social sciences.…”

Section: Introductionmentioning

confidence: 85%

“…The literature on the length of the citation window required to yield reliable information for measuring research output, especially at the departmental and institutional level, is large. For example, see Nederhof & van Raan, 1993;Adams, 2005;Van Leeuwen, 2006;Levitt & Thelwall, 2008, 2011Anderson & Tressler, 2013;Tressler & Anderson, 2012;Bertocchi et al, 2013;Wang, 2013;Stern, 2014;and Burns & Stern, 2015. In general, the findings provide some support for using direct citation counts to papers in the sciences, but little support for doing so in the social sciences. A notable exception is the recent work by Stern (2014) and Burns and Stern (2015).…”

Section: Introductionmentioning

confidence: 85%

Citation‐Capture Rates for Economics Journals: Do they Differ from Other Disciplines and Does it Matter?

J of App Econs and Policy

2015

Self Cite

This paper compares the rate of citation-capture across the social sciences, sciences, business school disciplines and economics. It also explores differences in the rate of citation-capture between leading journals in economics and a representative science category, and between higher and lower ranked economics journals. Short-term citation counting as used in impact factors and journal ranking schemes is shown to favour sciences over social sciences. In addition, within economics, short-term impact factors are systematically biased towards lower ranked journals. Our results imply that the use of leading multidisciplinary journal rankings as a proxy for direct citations introduces an unreported bias in research assessment. Such biases are in addition to those attributable to the well-known differences in the total number of cites received by papers across various disciplines.