Cumulated gain-based evaluation of IR techniques

Abstract

Modern large retrieval environments tend to overwhelm their users by their large output. Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation. In order to develop IR techniques in this direction, it is necessary to develop evaluation approaches and methods that credit IR methods for their ability to retrieve highly relevant documents. This can be done by extending traditional evaluation methods, that is, recall and precision based on binary relevance judgments, to graded relevance judgments. Alternatively, novel measures based on graded relevance judgments may be developed. This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. The first one accumulates the relevance scores of retrieved documents along the ranked result list. The second one is similar but applies a discount factor to the relevance scores in order to devaluate late-retrieved documents. The third one computes the relative-to-the-ideal performance of IR techniques, based on the cumulative gain they are able to yield. These novel measures are defined and discussed and their use is demonstrated in a case study using TREC data: sample system run results for 20 queries in TREC-7. As a relevance base we used novel graded relevance judgments on a four-point scale. The test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences. The graphs based on the measures also provide insight into the performance IR techniques and allow interpretation, for example, from the user point of view.

Adaptation of machine translation for multilingual information retrieval in the medical domain

“…The results of these experiments are reported in the first part of Table 19, using the standard IR evaluation measures: precision at a cut-off of 10 documents (P@10), normalized discounted cumulative gain [132] at 10 documents (N@10), and mean average precision (MAP) [133]. The cross-lingual MAP scores are also compared with the monolingual ones, i.e., those obtained by using the reference (English) translations of the test topics to see how the system would perform if the queries were translated perfectly (see columns denoted as MAP rel EN ).…”

Comparative Quality Estimation for Machine Translation Observations on Machine Learning and Features

“…2 Significance tests are based on the theoretical two-tailed t-test of tau and confidence intervals by bootstrap resampling (n = 1000, α = 0.05). NDCG is considered as an additional ranking metric (Järvelin and Kekäläinen, 2002).…”

Digital Resources for STEM Educators and Recommendations for Cyberlearning Initiatives: Results from the National Science Foundation Digital Library/Distributed Learning Program Evaluation

“…We also used another common measurement for search precision or relevance, cumulative gain, and its corollary, discounted cumulative gain. Cumulative gain is a standard strategy for assigning degrees of relevance search results (Dupret, 2011;Järvelin and Kekäläinen, 2002;Roelleke, 2013). 13 The NSDL portal uses Lucene, an open-source Apache search engine, as the core of its discovery service.…”