Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions

Abstract

Abstract. IR research has a strong tradition of laboratory evaluation of systems. Such research is based on test collections, pre-defined test topics, and standard evaluation metrics. While recent research has emphasized the user viewpoint by proposing user-based metrics and non-binary relevance assessments, the methods are insufficient for truly user-based evaluation. The common assumption of a single query per topic and session poorly represents real life. On the other hand, one well-known metric for multiple queries per session, instance recall, does not capture early (within session) retrieval of (highly) relevant documents. We propose an extension to the Discounted Cumulated Gain (DCG) metric, the Session-based DCG (sDCG) metric for evaluation scenarios involving multiple query sessions, graded relevance assessments, and open-ended user effort including decisions to stop searching. The sDCG metric discounts relevant results from later queries within a session. We exemplify the sDCG metric with data from an interactive experiment, discuss how the metric might be applied, and present research questions for which the metric is helpful.

Test Collection-Based IR Evaluation Needs Extension toward Sessions – A Case of Extremely Short Queries

“…Their information needs may initially be muddled and change during the search process; they may learn as the session progresses, or switch focus. The initial query formulation may not be optimal and the searchers may need to try out different wordings [2]. In fact, it may be impossible for the searcher to predict how well the query will perform [7] because even if the query describes the topic well, it may be ambiguous [8] and retrieve documents not serving the particular information need.…”

“…The stopping decisions regarding browsing the retrieved documents depend on the search task and the individual performing the task [2]. Jansen and colleagues [1] observed that most users did not access results beyond the first page, i.e., the top-10 results retrieved.…”

“…Järvelin et al [2] extended the Discounted Cumulated Gain metric [6] into a session-based evaluation metric which evaluates multiple query sessions and takes the searcher' s effort indirectly into account. Also the literature on usability has a comprehensive approach to costs and benefits, see, e.g., the ISO standard [15].…”

“…Recent studies show that real users of information retrieval (IR) systems search by very short queries but may try out several queries in a session [1][2][3][4][5]. Such queries may consist of only 1-2 search keys.…”

“…This setting is different from the Cranfield style IR experiments based on verbose queries and one query per topic. IR evaluation focuses on the quality of the ranked result, measured in terms of available single query metrics, such as mean average precision or cumulated gain or its variants [2,6]. These metrics do not directly pay attention to query formulation costs, that is, they encourage finding quality at any cost, and short queries are not rewarded for their minor formulation costs.…”