Publications as a measure of scientific advancement and of scientists' productivity

Daniel, Hans‐Dieter

doi:10.1087/0953151053584939

Cited by 47 publications

(30 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Cicchetti 9 defines inter-rater reliability 'as the extent to which two or more independent reviews of the same scientific document agree' (p. 120). Even though a high level of inter-rater reliability is often seen as desirable, the results of a meta-analysis for the inter-rater reliability of peer reviews for traditional print journals 10 confirm the findings of the narrative reviews published so far: 11,12 a low level of inter-rater reliability. (However, the study found also a statistically significant study-to-study variation of the inter-rater reliability coefficients.)…”

Section: Introductionsupporting

confidence: 53%

Reliability of reviewers' ratings when using public peer review: a case study

Bornmann

Daniel

2010

Learned Publishing

View full text Add to dashboard Cite

If a manuscript meets scientific standards and contributes to the advancement of science, it can be expected that two or more reviewers will agree on its value. Manuscripts are rated reliably when there is a high level of agreement between independent reviewers. This study investigates for the first time whether inter‐rater reliability, which is low with the traditional model of closed peer review, is also low with the new system of public peer review or whether higher coefficients can be found for public peer review. To investigate this question we examined the peer‐review process practiced by the interactive open access journal Atmospheric Chemistry and Physics (based on 465 manuscripts submitted between 2004 and 2006 receiving 1,058 reviews in total). The results of the study show that inter‐rater reliability is low (kappa coefficient) or reasonable (Intraclass Correlation Coefficient) in public peer review.

show abstract

Section: Introductionsupporting

confidence: 53%

Reliability of reviewers' ratings when using public peer review: a case study

Bornmann

Daniel

2010

Learned Publishing

View full text Add to dashboard Cite

show abstract

“…In a competitive environment where appointments, promotions, and grant applications are strongly influenced by publication and citation records, scientists are under intense pressure to publish (Daniel, 2005;Giles, 2005). Undoubtedly, academic careers are to a great extent related to number of publications and citations (Breitling, 2005;Klamer & van Dalen, 2002).…”

Section: Introductionmentioning

confidence: 99%

Multiple publication on a single research study: Does it pay? The influence of number of research articles on total citation counts in biomedicine

Bornmann

Daniel

2007

J. Am. Soc. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

Scientists may seek to report a single definable body of research in more than one publication, that is, in repeated reports of the same work or in fractional reports, in order to disseminate their research as widely as possible in the scientific community. Up to now, however, it has not been examined whether this strategy of "multiple publication" in fact leads to greater reception of the research. In the present study, we investigate the influence of number of articles reporting the results of a single study on reception in the scientific community (total citation counts of an article on a single study). Our data set consists of 96 applicants for a research fellowship from the Boehringer Ingelheim Fonds (BIF), an international foundation for the promotion of basic research in biomedicine. The applicants reported to us all articles that they had published within the framework of their doctoral research projects. On this single project, the applicants had published from 1 to 16 articles (M ‫؍‬ 4; Mdn ‫؍‬ 3). The results of a regression model with an interaction term show that the practice of multiple publication of research study results does in fact lead to greater reception of the research (higher total citation counts) in the scientific community. However, reception is dependent upon length of article: the longer the article, the more total citation counts increase with the number of articles. Thus, it pays for scientists to practice multiple publication of study results in the form of sizable reports.

show abstract

“…In particular, research evaluation is supposed to provide an effective signal to government agencies, sponsors, potential students, and their future employers about the reputation of research 1 For a detailed description of the evaluation process of management research in the United Kingdom see Bessant et al (2003) 2 For a defense of peer reviews see e.g. Daniel (2005).…”

Section: Performance Evaluations: Expectations and Disappointmentsmentioning

confidence: 99%

Evaluations: Hidden Costs, Questionable Benefits, and Superior Alternatives

Frey

Osterloh

2006

SSRN Journal

View full text Add to dashboard Cite

Research evaluation is praised as the symbol of modern quality management. We claim firstly, that performance evaluations in research have higher costs than normally assumed, because the evaluated persons and institutions systematically change their behavior and develop counter strategies. Moreover, intrinsic work motivation is crowded out and undesired lock-in effects take place. Secondly, the benefits of performance evaluations are questionable. Evaluations provide too little information relevant for decision-making. In addition, they lose importance due to new forms of scientific cooperation on the internet. Thirdly, there exist superior alternatives. They consist in careful selection and supportive process coaching -and then leave individuals and research institutions to direct themselves. (108 words)Key words: Evaluation, rankings, hidden costs, multi tasking, intrinsic motivation, control theory, selection.JEL Classification: C44, D02 ,D61, D72, H52, I23, M12 * The authors are also associated with CREMA -Center for Research in Economics, Management and the Arts, Zurich. 1 Performance Evaluations: Expectations and DisappointmentsPerformance evaluations have become a standard procedure in quality and performance management in many organizations. This procedure today is applied in profit-oriented firms, non-profit and governmental institutions. A particularly important area has been research evaluation on which this paper concentrates. Research evaluation is often praised as the symbol of modern quality management which today has diffused through the academic system. There are several academic journals (e.g."Evaluation", "Evaluation Quarterly", "Evaluation Review", "Research Evaluation", "Scientometrics") specifically devoted to the approach. Economics of evaluation has emerged as an academic discipline (e.g. citations, and co-citations as an answer to the growing skepticisms with peer reviews are also subject to major shortcomings, and can therefore not substitute for peer reviews with its own shortcomings (e.g. Üsdiken and Pasadeos 1995;Gmür 2003;Weingart 2005). These are important considerations but they have been extensively treated elsewhere. In this paper we focus on the hidden, and therefore often disregarded costs of performance evaluation.The hidden costs of evaluation consist of two main types, discussed in the first part of the paper: institutions. We argue in the second part of this paper that, firstly, evaluations provide too little information relevant for decision-making. Secondly, evaluations (in particular university rankings)lose importance due to new forms of scientific cooperation on the internet.The third part of the paper is devoted to alternatives to formal, retrospective and external evaluations of performance. They consist in(1) an interactive and supporting coaching of persons and institutions (concurrent process control).(2) a careful selection and socialization of persons charged with performing the activities in question (ex ante input control).This paper does not provi...

show abstract

Publications as a measure of scientific advancement and of scientists' productivity

Cited by 47 publications

References 10 publications

Reliability of reviewers' ratings when using public peer review: a case study

Reliability of reviewers' ratings when using public peer review: a case study

Multiple publication on a single research study: Does it pay? The influence of number of research articles on total citation counts in biomedicine

Evaluations: Hidden Costs, Questionable Benefits, and Superior Alternatives

Contact Info

Product

Resources

About