Open versus blind peer review: is anonymity better than transparency?

Shoham, Natalie; Pitman, Alexandra

doi:10.1192/bja.2020.61

Cited by 13 publications

(13 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…There has long been heated debate on how to reduce disparate review outcomes, with no agreement on the best approach [11][12][13] (Supplementary Table 1). A notorious example is the lack of consensus on the efficacy of double-blind peer review at eliminating bias from the review process.…”

Section: Possible Solutions To Reduce Biasmentioning

confidence: 99%

Peer review perpetuates barriers for historically excluded groups

et al. 2023

View full text Add to dashboard Cite

w reviewer homophily. These analyses leverage data from 31 studies, which cumulatively examined 312,740 manuscripts submitted to >640 journals-including Nature Portfolio journals, Science and Proceedings of the National Academy of Sciences. Our dataset represents 4,529,971 author position/demographic/review stage interactions, which upon publication will be the largest publicly available dataset of this kind for future work to build upon. Finally, we describe the current landscape of peer review in the subfields of ecology and evolution by collecting peer review policy data from the websites of 541 journals. Altogether, we find that author demographics predict review outcomes; clear, evidence-based solutions to alleviate review bias are lacking; and relatively few journals are pro-actively combatting bias. Results and discussion Disparate peer review outcomes by author demographicsWe found evidence for disparate peer review outcomes for all demographics that we examined at one or more stages in the review process (Supplementary Tables 3-40). We found the most data on outcomes by author assumed gender (Supplementary Data 1). Assumed female authors had worse or similar outcomes compared with assumed male authors, depending on the author position and review stage (Fig. 2).We found notably lower success throughout the review process for authors with institutional affiliations in Asia, in countries where English is not a primary language and in countries with lower Human Development Indices (HDI; Figs. 2 and 3). Compared with authors with affiliations located in Europe, North America and Oceania, authors with affiliations located in Asia had the most consistent disparities, but authors in Latin America and Africa also often had worse review outcomes. When considering a country's continent, language and HDI

show abstract

Section: Possible Solutions To Reduce Biasmentioning

confidence: 99%

Peer review perpetuates barriers for historically excluded groups

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Peer assessment can formally be defined as “an arrangement for learners to consider and specify the level, value, or quality of a product or performance of other equal‐status learners” (Topping, 2009, p. 20). Peer assessment can be incorporated under various settings such as online or in‐person (Yu & Wu, 2011), pairs or groups (Topping, 2010), open or blinded (Shoham & Pitman, 2021) on a wide range of learning activities including oral presentations (Nejad & Mahfoodh, 2019), essays (Huisman et al, 2018), portfolios (Yang et al, 2016), projects (Lin, 2018) and student‐generated content (Darvishi et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Incorporating AI and learning analytics to build trustworthy peer assessment systems

Darvishi

Khosravi

Sadiq

et al. 2022

Brit J Educational Tech

View full text Add to dashboard Cite

Peer assessment has been recognised as a sustainable and scalable assessment method that promotes higher‐order learning and provides students with fast and detailed feedback on their work. Despite these benefits, some common concerns and criticisms are associated with the use of peer assessments (eg, scarcity of high‐quality feedback from peer student‐assessors and lack of accuracy in assigning a grade to the assessee) that raise questions about their trustworthiness. Consequently, many instructors and educational institutions have been anxious about incorporating peer assessment into their teaching. This paper aims to contribute to the growing literature on how AI and learning analytics may be incorporated to address some of the common concerns associated with peer assessment systems, which in turn can increase their trustworthiness and adoption. In particular, we present and evaluate our AI‐assisted and analytical approaches that aim to (1) offer guidelines and assistance to student‐assessors during individual reviews to provide better feedback, (2) integrate probabilistic and text analysis inference models to improve the accuracy of the assigned grades, (3) develop feedback on reviews strategies that enable peer assessors to review the work of each other, and (4) employ a spot‐checking mechanism to assist instructors in optimally overseeing the peer assessment process. What is already known about this topic Engaging students in peer assessment has been demonstrated to have various benefits. However, there are some common concerns associated with employing peer assessment that raise questions about their trustworthiness as an assessment item. What this paper adds Methods and processes on how AI and learning analytics may be incorporated to address some of the common concerns associated with peer assessment systems, which in turn, can increase their trustworthiness and adoption. Implications for practice Presentation of a systematic approach for development, deployment and evaluation of AI and analytics approaches in peer assessment systems.

show abstract

“…Though some critics believe it also may result in inability to validate quality control (Das, 2016); open review is regularly compared with traditional double blind review. The efficacy of the new forms seems to be not much higher than that of double blind review (Shoham, & Pitman, 2020). But open peer review "increases accountability of reviewers".…”

Section: Peer Review Stakeholdersmentioning

confidence: 98%

Improving Submissions to Scholarly Journals via Peer Review

Tikhonova

Raitskaya

2021

JLE

View full text Add to dashboard Cite

Due to their commitment to better publishing standards and desire to improve their journals’ academic reputation, editorial boards, editors, and editorial teams seek to refine submissions they receive. Though, the peer review process serves as a filtering and assessment system, it is believed to greatly contribute to better quality of scholarly journals. Based on the analysis of the peer review internationally, the JLE editors focus on the peer review in the Journal of Language and Education, sharing their experience with the JLE potential authors. The editorial contains some reflections on the efficacy of peer review in the JLE. Potential authors may find some tips as to how to interact with recommendations and criticism on part of their peer reviewers and to make their voices heard.

show abstract

Open versus blind peer review: is anonymity better than transparency?

Cited by 13 publications

References 30 publications

Peer review perpetuates barriers for historically excluded groups

Peer review perpetuates barriers for historically excluded groups

Incorporating AI and learning analytics to build trustworthy peer assessment systems

Improving Submissions to Scholarly Journals via Peer Review

Contact Info

Product

Resources

About