When information retrieval measures agree about the relative quality of document rankings

Abstract

The variety of performance measures available for information retrieval systems, search engines, and network filtering agents can be confusing to both practitioners and scholars. Most discussions about these measures address their theoretical foundations and the characteristics of a measure that make it desirable for a particular application. In this work, we consider how measures of performance at a point in a search may be formally compared. Criteria are developed that allow one to determine the percent of time or conditions under which two different performance measures suggest that one document ordering is superior to another ordering, or when the two measures disagree about the relative value of document orderings. As an example, graphs provide illustrations of the relationships between precision and F.

Are two document clusters better than one? The Cluster Performance Question for information retrieval

“…Other performance measures could be used for such a study, or the relationship between performance with this measure and performance with other measures might be studied (Losee, 2000). An earlier study on distributed information retrieval provides methods using Average Search Length as a performance measure that can address issues such as the clustering problems discussed above (Losee & Church, 2004).…”

Is 1 noun worth 2 adjectives? Measuring relative feature utility

“…Determining the relative performance of two different characteristics used in ordering allows one to make decisions about which types of characteristics should be incorporated into a specific system. While ordering performance measures take many forms (Losee, 2000), e.g. precision, recall, average search length, or any of a number of measures of ordering performance, a linear measure that can be directly interpreted as an occurrence of characteristic X produces performance at the same level as occurrences of characteristic Y can be particularly valuable when making decisions about whether characteristic ¡ or characteristic ¢ should be used, along with their associated costs and benefits.…”

A geometric interpretation and analysis of R-precision

“…Since different evaluation measures evaluate different aspects of retrieval behavior, many evaluation measures have come to proliferate, and these measures have been deeply analyzed and criticized [11,8]. For example, Buckley and Voorhees compare evaluation measures based on their query stability [5], and Loose [10] proposes criteria to determine under which conditions different measures agree or disagree.…”