You Can Teach an Old Dog New Tricks: Rank Fusion applied to Coordination Level Matching for Ranking in Systematic Reviews

Abstract

Coordination level matching is a ranking method originally proposed to rank documents given Boolean queries that is now several decades old. Rank fusion is a relatively recent method for combining runs from multiple systems into a single ranking, and has been shown to significantly improve the ranking. This paper presents a novel extension to coordination level matching, by applying rank fusion to each sub-clause of a Boolean query. We show that, for the tasks of systematic review screening prioritisation and stopping estimation, our method significantly outperforms the state-of-the-art learning to rank and bag-ofwords-based systems for this domain. Our fully automatic, unsupervised method has (i) the potential for significant real-world cost savings (ii) does not rely on any intervention from the user, and (iii) is significantly better at ranking documents given only a Boolean query in the context of systematic reviews when compared to other approaches.

Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study

“…This ranking of studies has come to be known as 'screening prioritisation', as popularised by the CLEF TAR tasks which aimed to automate these early stages of the systematic review creation pipeline [9,11,10]. As a result, in recent years there has been an uptake in Information Retrieval approaches to enable screening prioritisation [18,5,3,25,2,22,16,15,1,27,21]. The vast majority of screening prioritisation use a different representation than the original Boolean query for ranking.…”

“…These studies are often referred to as 'seed studies' and are commonplace in the initial phases of the systematic review creation process. This method and others such as CLF [21] (which directly uses the Boolean query for ranking) have been shown to significantly outperform other methods that use a naïve query representation. Despite this, the SDR method was published when there was little data for those seeking to research this topic, and there have been methods published since that did not include SDR as a comparison.…”

Knowledge discovery from academic publications

“…Recently, Zou et al (2018); Zou and Kanoulas (2020) proposed an approach that looks for entities in the documents and asks questions to the users to retrieve the last relevant documents. Also, Scells et al (2020b) proposed a computational approach to objectively derive search strategies for systematic reviews and also presented a novel approach that ranks documents for systematic review literature using rank fusion applied to coordination level matching by taking advantage of the boolean query (Scells et al, 2020a).…”