The evaluation of scientific production: Towards a neutral impact factor

Abstract

Measurement of research activity still remains a controversial question. The use of the impact factor from the Institute for Scientific Information (ISI) is quite widespread nowadays to carry out evaluations of all kinds; however, the calculation formula employed by ISI in order to construct its impact factors biases the results in favour of knowledge fields which are better represented in the sample, cite more in average and whose citations are concentrated in the early years of the articles.In the present work, we put forward a theoretical proposal regarding how aggregated normalization should be carried out with these biases, which allows comparing scientific production between fields, institutions and/or authors in a neutral manner. The technical complexity of such work, together with data limitations, lead us to propose some adjustments on the impact factor proposed by ISI which -although they do not completely solve the problemreduce it and allow glimpsing the path towards more neutral evaluations. The proposal is empirically applied to three analysis levels: single journals, knowledge fields and the set of journals from the Journal Citation Report.

Citation characterization and impact normalization in bioinformatics journals

“…The RIF normalized the effect of article life span on the citation pattern. The impact of a journal's citing propensity also can be normalized by appending a new variable R i ( t ) to RIF (Gomez‐Sancho & Mancebón‐Torrubia, 2009; Zitt & Small, 2008). The propensity adjusted impact factor (PRIF) was computed for each journal by the following equation: …”

“…The citing propensity reflected by the number of references for the article can be normalized to compute the citation impact (Biglu, 2008; Gomez‐Sancho & Mancebón‐Torrubia, 2009). Specifically, fractional counting (Small & Sweeney, 1985) was used for co‐citation analysis and incorporated into journal normalization (Leydesdorff & Bornmann, 2011; Moed, 2010; Zitt & Small, 2008).…”

A CERIF data model extension for evaluation and quantitative expression of scientific research results

“…Moreover, those data about results can be a starting point for various statistical examinations related to science. Papers (Schubert 2002;Qiu and Chen 2009;Gómez-Sancho and Mancebón-Torrubia 2009;Molatudi et al 2009;Klitkou and Gulbrandsen 2010) for the starting point of their research use data from research management systems. Also, those data can be used for evaluation of scientific research results according to national, regional and international rule books for evaluation and quantitative expression of scientific research results.…”

Evidence base, quantitation and collaboration: three novel indices for bibliometric content analysis

“…They tell us nothing about the characteristics of the research product, nor about the nature of the researchers themselves. More recent bibliometric studies have sought to address these biases (Ball et al 2009;Gomez-Sancho and Mancebon-Torrubia 2009), while others have directed attention to the actual production and characteristics of research. Bibliometric studies have reported on methodologies and evidence base using research design hierarchies (Papatheodorou et al 2008;Rochon et al 2004); recognised recommendations (Yoshii et al 2009), and grading systems (Brooke et al 2009;Moseley et al 2002).…”