Novel tools to streamline the conference review process

Abstract

The SIGKDD'09 Research Track received 537 paper submissions, which were reviewed by a Program Committee of 199 members, and a Senior Program Committee of 22 members. We used techniques from artificial intelligence and data mining to streamline and support this complicated process at three crucial stages: bidding by PC members on papers, assigning papers to reviewers, and calibrating scores obtained from the reviews. In this paper we report on the approaches taken, evaluate how well they worked, and describe some further work done after the conference.

An application of hierarchical Gaussian processes to the detection of anomalies in star light curves

“…In so doing, model inference can be expedited, model interpretability can be increased and models can be fairly compared for model selection since bespoke architectures have been crafted for the problem [26]. It is often non-trivial to encode this domain knowledge into the language of graphical models, but when successful this can produce elegant and accurate models in many domain areas, including skill assessment [27], matching [26], reviewing [28] and recommendation systems [29]. Indeed, this is the key idea that has recently inspired a revolution in probabilistic programming frameworks in the machine learning field [30,31,32,33] that offer a high-level interface for modelling and factorising data.…”

Computational support for academic peer review

“…14 A range of methods have been used to reduce the human effort involved in paper allocation, typically with the aim of producing assignments that are similar to the 'gold standard' manual process. 9,13,16,18,30,34,37 Yet, despite many publications on this topic over the intervening years, research results in paper assignment have made relatively few inroads into mainstream CMS tools and everyday peer review practice. Hence, what we have achieved over the last 25 years or so appears to be a streamlined process rather than a fundamentally improved one: we believe it would be difficult to argue the decisions taken by program committees today are significantly better in comparison with the paper-based process.…”

Calibration with confidence: a principled method for panel assessment

“…We are aware of two other schemes that incorporate confidences into a calibration process. One is the abstract-review method for the SIGKDD'09 conference (section 4 of [6]; see also [7]). The other is the abstract-review method used for the NIPS2013 conference (building on [8] and described in [9]).…”

“…as a shorthand for the definitions in equations (8) and (9), so that the equations (6,7) can be written as…”