Bias and the limits of pooling for large collections

Abstract

Modern retrieval test collections are built through a process called pooling in which only a sample of the entire document set is judged for each topic. The idea behind pooling is to find enough relevant documents such that when unjudged documents are assumed to be nonrelevant the resulting judgment set is sufficiently complete and unbiased. Yet a constant-size pool represents an increasingly small percentage of the document set as document sets grow larger, and at some point the assumption of approximately complete judgments must become invalid. This paper shows that the judgment sets produced by traditional pooling when the pools are too small relative to the total document set size can be biased in that they favor relevant documents that contain topic title words. This phenomenon is wholly dependent on the collection size and does not depend on the number of relevant documents for a given topic. We show that the AQUAINT test collection constructed in the recent TREC 2005 workshop exhibits this biased relevance set; it is likely that the test collections based on the much larger GOV2 document set also exhibit the bias. The paper concludes with suggested modifications to traditional pooling and evaluation methodology that may allow very large reusable test collections to be built.

On run diversity in Evaluation as a Service

“…Test collections are research tools that are most useful when they are reusable, that is, when they fairly evaluate retrieval systems that did not contribute to their construction [1]. Thus, a reusable test collection can be used to evaluate systems even after the original evaluation that created the collection has concluded.…”

“…Diversity is important from at least two perspectives. Previous work has tied the reusability of a collection built using pooling to the diversity of the retrieval runs that form the pools [1]. Diversity in participant submissions also serves as a proxy indicator that researchers are trying different (novel) retrieval techniques.…”

Score adjustment for correction of pooling bias

“…In addition, the bias can be systematic against systems that are different in nature from those which contributed to the pool. For instance, Buckley et al [2007] suggest that recent large TREC collections have their pools flooded with documents rich in query keywords, and are biased against retrieval methods that attempt to go beyond keyword matching. Such systematic bias is not merely unfair to certain systems, but is an obstacle to an entire direction of potential retrieval improvement.…”

Importance sampling for unbiased on-demand evaluation of knowledge base population

“…However, when the topic was later revisited, Buckley et al (2007) identified that the reason for the small bias was because the submissions to the task were too similar; upon repeating the experiment using a novel system as part of the TREC Robust track, they identified a 23% point drop in average precision scores! 8…”