IR evaluation methods for retrieving highly relevant documents

Abstract

This paper proposes evaluation methods based on the use of non-dichotomous relevance judgements in IR experiments. It is argued that evaluation methods should credit IR methods for their ability to retrieve highly relevant documents. This is desirable from the user point of view in modem large IR environments. The proposed methods are (1) a novel application of P-R curves and average precision computations based on separate recall bases for documents of different degrees of relevance, and (2) two novel measures computing the cumulative gain the user obtains by examining the retrieval result up to a given ranked position. We then demonstrate the use of these evaluation methods in a case study on the effectiveness of query types, based on combinations of query structures and expansion, in retrieving documents of various degrees of relevance. The test was run with a best match retrieval system (In- Query I) in a text database consisting of newspaper articles. The results indicate that the tested strong query structures are most effective in retrieving highly relevant documents. The differences between the query types are practically essential and statistically significant. More generally, the novel evaluation methods and the case demonstrate that non-dichotomous relevance assessments are applicable in IR experiments, may reveal interesting phenomena, and allow harder testing of IR methods.

Intelligent Social Networks Model Based on Semantic Tag Ranking

“…The other two is for evaluating ranking and these metrics are mean average precision (MAP) and Normalized Discounted Cumulative Gain (NDCG) [13].…”

TopPPR

“…Following previous work [30,31,39,40], we set the decay factor α of PPR to 0.2. We evaluate the accuracy of each method using two classic metrics for evaluating ranking results: precision and Normalized Discounted Cumulative Gain (NDCG) [25]. Specifically, given a query node s, let V k = {t 1 , .…”

A Serendipity-Oriented Greedy Algorithm for Recommendations

“…• To measure a ranking ability of an algorithm, we use normalized discounted cumulative gain (NDCG), which, in turn, is based on discounted cumulative gain (DCG) (Järvelin and Kekäläinen, 2000):…”