geNov: A new metric for measuring novelty and relevancy in biomedical information retrieval

Abstract

For diversity and novelty evaluation in information retrieval, we expect that the novel documents are always ranked higher than the redundant ones and the relevant ones higher than the irrelevant ones. We also expect that the level of novelty and relevancy should be acknowledged. Accordingly, we expect that the evaluation algorithm would reward rankings that respect these expectations. Nevertheless, there are few research articles in the literature that study how to meet such expectations, even fewer in the field of biomedical information retrieval. In this article, we propose a new metric for novelty and relevancy evaluation in biomedical information retrieval based on an aspect-level performance measure introduced by TREC Genomics Track with formal results to show that those expectations above can be respected under ideal conditions. The empirical evaluation indicates that the proposed metric, geNov, is greatly sensitive to the desired characteristics above, and the three parameters are highly tuneable for different evaluation preferences. By experimentally comparing with state-of-the-art metrics for novelty and diversity, the proposed metric shows its advantages in recognizing the ranking quality in terms of novelty, redundancy, relevancy, and irrelevancy and in its discriminative power. Experiments reveal the proposed metric is faster to compute than state-of-the-art metrics.

Three approaches to measuring recall on the Web: a systematic review

“…(1) Sampath Kumar and Prakash, 2009;Stokes, Foster and Urquhart, 2009; (3) Croft, Metzler and Strohman, 2010;Deka and Lahkar, 2010;Magdy and Jones, 2010;Sampath Kumar and Pavithra, 2010; (3) Ali and Beg, 2011;Levene, 2011;Teixeira Lopes and Ribeiro, 2011; (5) Baccini, Déjean, Lafage and Mothe, 2012;Bilal, 2012;Carterette, Kanoulas and Yilmaz, 2012;Li, Wang and Yu, 2012;Zuva and Zuva, 2012; (3) Clough and Sanderson, 2013;Kumar and Bhadu, 2013;Sirotkin, 2013; (3) Bama, Ahmed and Saravanan, 2015;Frants, Shapiro and Voiskunskii, 2015;Saracevic, 2015; (3) Ali and Gul, 2016;Arora, Kanjilal and Varshney, 2016;Wani Zahid and Ahmad Sofi, 2016; (3) An and Huang, 2017;Buckland, 2017;Buckley and Voorhees, 2017;…”

JASIST special issue on biomedical information retrieval

“…An and Huang (), in a paper titled “geNov: A New Metric for Measuring Novelty and Relevancy in Biomedical Information Retrieval,” deal with the importance of similarity measure for retrieval. In this paper, the authors propose a new metric for novelty and relevancy evaluation in biomedical information retrieval based on an aspect‐level performance measure introduced by TREC Genomics Track with formal results to show that those expectations above can be respected under ideal conditions.…”