On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications

Abstract

In reference to the increasing significance of citation counting in evaluations of scientists and science institutes as well as in science historiography, it is analyzed empirically what is cited in which frequency and what types of citations in scientific texts are used. Content analyses refer to numbers of references, self-references, publication language of references cited, publication types of references cited, and type of citation within the texts. Validity of citation counting is empirically analyzed with reference to random samples of English and German journal articles as well as German textbooks, encyclopedias, and test-manuals from psychology. Results show that 25% of all citations are perfunctory, more than 50% of references are journal articles and up to 40% are books and book-chapters, 10% are self-references. Differences between publications from various psychological sub-disciplines, publication languages, and types of publication are weak. Thus, validity of evaluative citation counting is limited because at least one quarter refers to perfunctory citations exhibiting a very low information utility level and by the fact that existing citationdatabases refer to journal articles only.

The Use of Bibliometrics to Measure Research Performance in Education Sciences

“…The statistics based on the number of publications primarily reflect the quantitative output of research activity. In contrast, there is little agreement on what the figures based on citations exactly measure, as the reasons for citing a paper may be highly disparate (see Jokić and Ball 2006, Krampen et al 2007, Moed 2005. 6 This paper 5 Because Web of Science only includes publications that have been published in scientific journals that are listed in the Social Sciences Citation Index, even a purely quantitative analysis of these publications implies a qualitative element, as the vast majority of included publications will have been subjected to review prior to acceptance for publication.…”

Citation rates for experimental psychology articles published between 1950 and 2004: Top-cited articles in behavioral cognitive psychology

“…(For example, an h-index of 5 indicates that the researcher has five articles, each of which has accrued at least five citations.) Despite the limitations of the h-index (see Krampen, Becker, Wahner, & Montada, 2007), at academic institutions emphasizing research, it is currently one of the most widely used measures (along with grant dollars received) for determining whether a candidate should be hired, tenured, or promoted, because it is a composite measure of both quantitative research output (i.e., the number of articles one has produced) and the impact that those articles have had on guiding the research of others.…”

“…(For example, an h-index of 5 indicates that the researcher has five articles, each of which has accrued at least five citations.) Despite the limitations of the h-index (see Krampen, Becker, Wahner, & Montada, 2007), at academic institutions emphasizing research, it is currently one of the most widely used measures (along with grant dollars received) for determining whether a candidate should be hired, tenured, or promoted, because it is a composite measure of both quantitative research output (i.e., the number of articles one has produced) and the impact that those articles have had on guiding the research of others.Another way that citations can be used is to determine which individual publications have had a major impact on guiding the research of others in that research area. For cognitive psychology, White (1983) identified 50 prominent publications (journal articles and books) indexed in the 1979-1982 volumes of the Social Sciences Citation Index (SSCI) that (a) had averaged more than 20 citations per year in the 1971-1982 SSCI and had at least five citations per year in any one of the eight major journals he surveyed.…”

Mathematical Approaches to Modeling Science from an Algorithmic-Historiography Perspective

“…Nevertheless, different studies have agreed that around 70% of the references used in a scientific paper correspond to criteria of scientific relevance (Vinkler 1996;Krampen et al 2007). In other words, 70% of citations respond to the normative theory of citing (Cronin 1984), which justifies the value of citation analysis for historical reconstruction of scientific fields.…”