When nonreliability of reviews indicates solid science

Eckberg, Douglas Lee

doi:10.1017/s0140525x00065791

Cited by 23 publications

(18 citation statements)

References 235 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…According to J. R. Cole (2000, p. 115), a low level of agreement among reviewers reflects the lack of consensus that is prevalent in all scientific disciplines at the “research frontier.” Cole says that usually no one reliably assesses scientific work occurring at the frontiers of research. Eckberg (1991) and Kostoff (1995) point out that differing judgments in peer review are not necessarily a sign of disagreement about the quality of a manuscript but may instead reveal differing positions, judgment criteria, and areas of competency among the reviewers. In addition, reviewers tend to be either more critical or more lenient in their judgments (Siegelman, 1991) if they direct their attention to “different points, and may draw different conclusions about ‘worth’” (Eckberg, 1991, p. 146).…”

Section: Reliability Fairness and Predictive Validity Of Peer Reviewmentioning

confidence: 99%

Scientific peer review

Bornmann

2011

Annual Review Info Sci & Tec

404

329

View full text Add to dashboard Cite

Section: Reliability Fairness and Predictive Validity Of Peer Reviewmentioning

confidence: 99%

Scientific peer review

Bornmann

2011

Annual Review Info Sci & Tec

404

329

View full text Add to dashboard Cite

“…While peer review is still generally perceived as key to quality control for research, it has been argued that mistakes are becoming more frequent in the process ( Margalida & Colomer, 2016; Smith, 2006), and that peer review is not being applied as rigorously as generally perceived. As a result, it has become the target of widespread criticism, with a range of empirical studies investigating the reliability, credibility and fairness of the scholarly publishing and peer review process (e.g., ( Bruce et al , 2016; Cole, 2000; Eckberg, 1991; Ghosh et al , 2012; Jefferson et al , 2002; Kostoff, 1995; Ross-Hellauer, 2017; Schroter et al , 2006; Walker & Rocha da Silva, 2015)). In response to this, initiatives like the EQUATOR network ( equator-network.org) have been important to improve the reporting of research and its peer review according to standardised criteria.…”

Section: 01 Methodsmentioning

confidence: 99%

A multi-disciplinary perspective on emergent and future innovations in peer review

et al. 2017

View full text Add to dashboard Cite

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

show abstract

“…A discrepancy in or deviating evaluation of papers is, however, not necessarily a sign of poor-quality contributions (see Eckberg 1991).…”

Section: Standardization Allowing For More Than One Viewmentioning

confidence: 99%