A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions

Stelmakh, Ivan; Rastogi, Charvi; Shah, Nihar B.; Singh, Aarti; Daumé, Hal

doi:10.48550/arxiv.2011.15083

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Within the machine learning community, [31] survey reviewers about visibility of papers submitted to a conference that anonymizes authors, and intentionally searching online for assigned papers. Or current work contributes to a tradition in machine learning venues of experimentation aimed at understanding and improving the peer-review process [3,5,16,[32][33][34][35][36][37][38].…”

Section: Related Workmentioning

confidence: 99%

How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions?

Rastogi,

Stelmakh,

Beygelzimer

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we surveyed the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews. The salient results are: (1) Authors had roughly a three-fold overestimate of the acceptance probability of their papers: The median prediction was 70% for an approximately 25% acceptance rate. (2) Female authors exhibited a marginally higher (statistically significant) miscalibration than male authors; predictions of authors invited to serve as meta-reviewers or reviewers were similarly calibrated, but better than authors who were not invited to review. (3) Authors’ relative ranking of scientific contribution of two submissions they made generally agreed with their predicted acceptance probabilities (93% agreement), but there was a notable 7% responses where authors predicted a worse outcome for their better paper. (4) The author-provided rankings disagreed with the peer-review decisions about a third of the time; when co-authors ranked their jointly authored papers, co-authors disagreed at a similar rate—about a third of the time. (5) At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process. The stakeholders in peer review should take these findings into account in setting their expectations from peer review.

show abstract

Section: Related Workmentioning

confidence: 99%

How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions?

Rastogi,

Stelmakh,

Beygelzimer

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…One motivation for our work is that of running controlled experiments in peer review. Controlled experiments pertaining to peer review are conducted in many different scientific fields (Armstrong 1980;Pier et al 2017;Teplitskiy et al 2019;Ceci and Peters 1982;Patat et al 2019), including several controlled experiments recently conducted in computer science (Lawrence 2014;Tomkins, Zhang, and Heavlin 2017;Stelmakh et al , 2020. These experiments have also led to a relatively nascent line of work on careful design of experimental methods for peer review Singh 2019b, 2021), and our work sheds some light in this direction in terms of trading off assignment quality with randomization in the assignment.…”

Section: Related Workmentioning

confidence: 99%

Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

Steven

Zhang

Liu

et al. 2022

HCOMP

View full text Add to dashboard Cite

Many scientific conferences employ a two-phase paper review process, where some papers are assigned additional reviewers after the initial reviews are submitted. Many conferences also design and run experiments on their paper review process, where some papers are assigned reviewers who provide reviews under an experimental condition. In this paper, we consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity? We make several contributions towards answering this question. First, we prove that when the set of papers requiring additional review is unknown, a simplified variant of this problem is NP-hard. Second, we empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment. This uniformly random choice is practical for both the two-phase and conference experiment design settings. Third, we provide explanations of this phenomenon by providing theoretical bounds on the suboptimality of this random strategy under certain natural conditions. From these easily-interpretable conditions, we provide actionable insights to conference program chairs about whether a random reviewer split is suitable for their conference.

show abstract

“…Stelmakh et al (2021b) study a pipeline for getting new reviewers into the review pool. Stelmakh et al (2020) study herding in discussions. A number of recent works (Charlin and Zemel, 2013;Stelmakh et al, 2021a;Kobren et al, 2019;Jecmen et al, 2020;Noothigattu et al, 2021) have designed algorithms that are used in the peer-review process of various computer science conferences.…”

Section: Related Workmentioning

confidence: 99%

To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online

Rastogi¹,

Stelmakh²,

Shen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Double-blind conferences have engaged in debates over whether to allow authors to post their papers online on arXiv or elsewhere during the review process. Independently, some authors of research papers face the dilemma of whether to put their papers on arXiv due to its pros and cons. We conduct a study to substantiate this debate and dilemma via quantitative measurements. Specifically, we conducted surveys of reviewers in two top-tier double-blind computer science conferences-ICML 2021 (5361 submissions and 4699 reviewers) and EC 2021 (498 submissions and 190 reviewers). Our two main findings are as follows. First, more than a third of the reviewers self-report searching online for a paper they are assigned to review. Second, outside the review process, we find that preprints from better-ranked affiliations see a weakly higher visibility, with a correlation of 0.06 in ICML and 0.05 in EC. In particular, papers associated with the top-10-ranked affiliations had a visibility of approximately 11% in ICML and 22% in EC, whereas the remaining papers had a visibility of 7% and 18% respectively.

show abstract

A Large Scale Randomized Controlled Trial on Herding in Peer-Review Discussions

Cited by 4 publications

References 0 publications

How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions?

How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions?

Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online

Contact Info

Product

Resources

About