Conference reviewing considered harmful

Abstract

This paper develops a model of computer systems research to help prospective authors understand the often obscure workings of conference program committees. We present data to show that the variability between reviewers is often the dominant factor as to whether a paper is accepted. We argue that paper merit is likely to be zipf distributed, making it inherently difficult for program committees to distinguish between most papers. We use game theory to show that with noisy reviews and zipf merit, authors have an incentive to submit papers too early and too often. These factors make conference reviewing, and systems research as a whole, less efficient and less effective. We describe some recent changes in conference design to address these issues, and we suggest some further potential improvements.

Network Information Hiding and Science 2.0: Can it be a Match?

“…Bücheler and Sieg study the applicability of Crowdsourcing and Open Innovation in the scientific context [8] while West et al [9] review the contribution and evolution of Open Innovation from the history perspective. Franzoni [12]. We will refer to selected aspects of his work in the context of network information hiding in Section V-3.…”

“…Anderson suggests to improve the transparency of the review process in [12]. One of his suggestions is that all reviewed papers of a conference should be published online.…”

“…Without Science 2.0, accepted papers appear in proceedings and journals while rejected paper will not appear although they can still contribute significantly to research. For this reason, Anderson states that the research community is worse off if rejected papers remain unpublished [12]. This argument is supported by Mogul's statement that computer science reviewing is becoming increasingly "hypercritical" [30].…”

“…In network information hiding, for instance, some submitted papers lack a strong evaluation or a proof of concept implementation. If even rejected submissions will be published online, authors may not want to see their names on their own publications as these are unfinished [12]. In this context, Science 2.0 can increase the quality of submissions by preventing intentionally lowqualified papers which are submitted only to receive review feedback.…”

“…Instead of publishing unfinished work, ongoing work can be developed online together with other scientists before it will be submitted. Overall, such approaches, including the community review of web publications [12], [30], could speed up the progress in information hiding while also easing progress tracking. 5) Required Effort: As mentioned in Section V-3, Science 2.0 tools enable a faster tracking of the domain's progress.…”