An evolutionary explanation of assassins and zealots in peer review

Chamorro-Padial, Jorge; Rodrı́guez-Sánchez, Rosa; Fdez-Valdivia, J.; García, J. A.

doi:10.1007/s11192-019-03171-3

Cited by 4 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This approach also does not allow for assessment of a reviewer's rating style, for example the tendency to severe/lenient assessments, which, as has been presented in many studies, influences the final evaluation [16,20,36]. Discordant evaluations among referees could make the review system unfair to authors whose manuscripts happened to be sent to an assassin or zealot reviewer [42] and thus should be controlled.…”

Section: Resultsmentioning

confidence: 99%

Ensuring Sustainable Evaluation: How to Improve Quality of Evaluating Grant Proposals?

Wieczorkowska

Kowalczyk

2021

Sustainability

View full text Add to dashboard Cite

The principle of sustainable development is an obligation placed on all entities involved in the implementation and delivery of the structural funds made available not only by the European Commission but also by grant donors from all over the world. For this reason, when applying for a grant, proposals need to demonstrate the positive or neutral impact of the project on sustainable development. To be able to select projects that will ensure sustainability, we need to ensure the effective evaluation of the proposals. The process of their evaluation should be objective, unbiased and transparent. However, current processes have several limitations. The process by which grants are awarded and proposals evaluated has come under increasing scrutiny, with a particular focus on the selection of reviewers, fallibility of their assessments, the randomness of assessments and the low level of common agreement. In our studies, we demonstrated how some of those limitations may be overcome. Our topic of interest is the work of reviewers/experts who evaluate scientific grant proposals. We analyse data coming from two prominent scientific national grant foundations, which differ in terms of expert’s selection procedure. We discuss the problems associated with both procedures (rating style of the reviewers, lack of calibration and serial position effect) and present potential solutions to prevent them. We conclude that, to increase the unbiasedness and fairness of the evaluation process, reviewers’ work should be analysed. We also suggest that, within a certain panel, all grant proposals should be evaluated by the same set of reviewers, which would help to eliminate the distorting influence of the selection of a very severe or very lenient expert. Such effective assessment and moderation of the process would help ensure the quality and sustainability of evaluations.

show abstract

Section: Resultsmentioning

confidence: 99%

Ensuring Sustainable Evaluation: How to Improve Quality of Evaluating Grant Proposals?

Wieczorkowska

Kowalczyk

2021

Sustainability

View full text Add to dashboard Cite

show abstract

“…The same concept is applied to reviewers [5]. The quasi-species model inspires our work to determine the evolution of an authors' profiles after the peer-review process.…”

Section: State Of the Art And Related Workmentioning

confidence: 99%

“…The evolutionary success of a quasi-species strongly depends on the replication rates of clouds. In [5] the authors adapted the quasi-species model from biology to the author-editor game's evolutionary environment. Self-replicating entities are submission profiles under a given partition of manuscript categories.…”

Section: State Of the Art And Related Workmentioning

confidence: 99%

The relevance of title, abstract, and keywords for scientific paper quality and potential impact

Chamorro-Padial

Rodrı́guez-Sánchez

2023

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

Authors, editors, and reviewers need to have a good perception regarding the quality of a manuscript in order to improve their skills, save effort, and prevent errors that can affect the submission procedure. In this paper, we compared the author’s perception of a manuscript’s quality with the manuscript’s actual impact. In addition, we analyzed the uncertainty of the author’s perception of the manuscript’s quality. From there, we defined ‘partition’ as the author’s ability to perceive the actual quality. We did this by launching a website for the use of the scientific community. This webpage provided a tool to help improve an investigator’s skill in understanding and recognizing the quality of a manuscript so as to help researchers improve and maximize their works’ potential impact. We carried out the experiment with 106 experienced users who tested our webpage. We found that the Abstract, the Title, and the Keywords were enough to perform a substantially decent evaluation of a manuscript. Most of the researchers were able to determine the quality of a paper in less than a minute from this small amount of information.

show abstract

“…Grimaldo et al [36] used ABM to simulate the behaviour of journals receiving highquality articles, and especially discussed the influence of outliers on peer review and journal word-of-mouth. Garcı ´a's and Chamorro-Padial's team [37][38][39] published papers three times, analysing how editors' preferences affect contributors in 2015, explaining why assassins and zealots evolutionary appear in peer review in 2019, and discussing the shortcomings of peer review incentive system in 2020. They [39] synthesised various problems in peer review and tried to design a system to encourage high-quality peer review.…”

Section: Scientometricsmentioning

confidence: 99%

“…Chamorro-Padial et al [38] divided the behaviour patterns of contributors into assassins and zerots, and discussed the impact of editing preferences on the behaviour of contributors. Knowledge Management Provide potential solutions to problems that are difficult to quantify the concept of 'knowledge'.…”

Section: Scientometricsmentioning

confidence: 99%

Agent-based model: A method worthy of promotion in Library and Information Science

Chen

Wang

2022

Journal of Information Science

View full text Add to dashboard Cite

Agent-based model (ABM) is a branch of artificial intelligence. Its specialty is to construct a complex macro-system model by describing the perception, decision, learning and action of micro-agents. This method is widely used in many fields from natural science to social science. We discuss ABM by collecting relevant academic papers which apply to the field of Library and Information Science (LIS). This article systematically reviews how ABM is applied to the LIS field and argues that ABM can provide an exploratory tool with quantifiability, repeatability, interpretability, contingency, adaptability and other types of advantages. Finally, it is pointed out that this method is a research tool worthy of careful exploration.

show abstract

An evolutionary explanation of assassins and zealots in peer review

Cited by 4 publications

References 15 publications

Ensuring Sustainable Evaluation: How to Improve Quality of Evaluating Grant Proposals?

Ensuring Sustainable Evaluation: How to Improve Quality of Evaluating Grant Proposals?

The relevance of title, abstract, and keywords for scientific paper quality and potential impact

Agent-based model: A method worthy of promotion in Library and Information Science

Contact Info

Product

Resources

About