Citation Analysis May Severely Underestimate the Impact of Clinical Research as Compared to Basic Research

Abstract

BackgroundCitation analysis has become an important tool for research performance assessment in the medical sciences. However, different areas of medical research may have considerably different citation practices, even within the same medical field. Because of this, it is unclear to what extent citation-based bibliometric indicators allow for valid comparisons between research units active in different areas of medical research.MethodologyA visualization methodology is introduced that reveals differences in citation practices between medical research areas. The methodology extracts terms from the titles and abstracts of a large collection of publications and uses these terms to visualize the structure of a medical field and to indicate how research areas within this field differ from each other in their average citation impact.ResultsVisualizations are provided for 32 medical fields, defined based on journal subject categories in the Web of Science database. The analysis focuses on three fields: Cardiac & cardiovascular systems, Clinical neurology, and Surgery. In each of these fields, there turn out to be large differences in citation practices between research areas. Low-impact research areas tend to focus on clinical intervention research, while high-impact research areas are often more oriented on basic and diagnostic research.ConclusionsPopular bibliometric indicators, such as the h-index and the impact factor, do not correct for differences in citation practices between medical fields. These indicators therefore cannot be used to make accurate between-field comparisons. More sophisticated bibliometric indicators do correct for field differences but still fail to take into account within-field heterogeneity in citation practices. As a consequence, the citation impact of clinical intervention research may be substantially underestimated in comparison with basic and diagnostic research.

Field-normalized citation impact indicators using algorithmically constructed classification systems of science

“…In Van Eck et al (2013), the authors had a priori information about this possibility in a number of subject categories within the WoS system. The problem, of course, is that we do not have any information about which clusters may lack the desirable homogeneity within any given classification system.…”

“…On the other hand, using the 20 sections of the Biochemistry heading they clearly illustrate that citation habits vary extensively not only between fields but also within fields. Similarly, Van Eck et al (2013) establish the existence of heterogeneous sub-groups (corresponding to clinical and basic medical research) with different citation practices within WoS subject categories. In this case, the mean citation of an entire category is simply the weighted average of different, and hence non-comparable sub-group mean citations.…”

A Bibliometric Review of Research Trends in Neuroimaging

“…Moreover, it has been reported that researches within the same field have received different extents of attention and thus the number of citations. For example, basic researches had a higher impact than clinical interventional researches 9 . Therefore, the aim of this study is to describe the general landscape of the neuroimaging literature, evaluate performance trends of neuroimaging journals as a whole and to analyse four representative journals with the highest and lowest impact factors (IFs) within this category.…”

Bridging the Evaluation Gap

“…This was partly motivated by the simple and rather old-fashioned criterion of consistency: if we are learning things in our research that have bearing on the accuracy and/or legitimacy of the evaluation process, we should bring that knowledge to the evaluation itself rather than pretend that the customary/existing evaluation method is sufficient to realizing the objectives that the evaluators themselves bring to the process. For example, our bibliometric research had shown that citation densities differ considerably within disciplines and the usual normalization of indicators does not correct for those differences (van Eck et al 2013). Because of this, we felt we couldn't really make the claim that the relatively high citation scores of the bibliometric research at our center was in itself already evidence of its high quality.…”