Performance standards and evaluations in IR test collections: Vector-space and other retrieval models

Shaw, W. M.; Burgin, Robert; Howell, Patrick

doi:10.1016/s0306-4573(96)00044-1

Cited by 30 publications

(22 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The univariate central hypergeometric distribution has been firstly used in the past to provide a theoretical framework for understanding performance and evaluation measures in IR [11,35], and for proving the document-query duality [10].…”

Section: Related Workmentioning

confidence: 99%

Hypergeometric language models for republished article finding

Tsagkias

Rijke

Weerkamp

2011

Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Republished article finding is the task of identifying instances of articles that have been published in one source and republished more or less verbatim in another source, which is often a social media source. We address this task as an ad hoc retrieval problem, using the source article as a query. Our approach is based on language modeling. We revisit the assumptions underlying the unigram language model taking into account the fact that in our setup queries are as long as complete news articles. We argue that in this case, the underlying generative assumption of sampling words from a document with replacement, i.e., the multinomial modeling of documents, produces less accurate query likelihood estimates.To make up for this discrepancy, we consider distributions that emerge from sampling without replacement: the central and noncentral hypergeometric distributions. We present two retrieval models that build on top of these distributions: a log odds model and a bayesian model where document parameters are estimated using the Dirichlet compound multinomial distribution.We analyse the behavior of our new models using a corpus of news articles and blog posts and find that for the task of republished article finding, where we deal with queries whose length approaches the length of the documents to be retrieved, models based on distributions associated with sampling without replacement outperform traditional models based on multinomial distributions.

show abstract

Section: Related Workmentioning

confidence: 99%

Hypergeometric language models for republished article finding

Tsagkias

Rijke

Weerkamp

2011

Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

show abstract

“…The E measure is computed as E ϭ 1 Ϫ 2/(P Ϫ1 ϩ R Ϫ1 ), where P is the precision and R is the recall. The F measure is computed as 1 Ϫ E and has the attractive feature that high values represent better performance than lower values, the opposite of the situation with the E measure (Shaw et al, 1997).…”

Section: Retrieval and Filtering Measuresmentioning

confidence: 99%

When information retrieval measures agree about the relative quality of document rankings

Losee

2000

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

The variety of performance measures available for information retrieval systems, search engines, and network filtering agents can be confusing to both practitioners and scholars. Most discussions about these measures address their theoretical foundations and the characteristics of a measure that make it desirable for a particular application. In this work, we consider how measures of performance at a point in a search may be formally compared. Criteria are developed that allow one to determine the percent of time or conditions under which two different performance measures suggest that one document ordering is superior to another ordering, or when the two measures disagree about the relative value of document orderings. As an example, graphs provide illustrations of the relationships between precision and F.

show abstract

“…In general, an IR model specifies a document representation, a query representation and a matching procedure. The majority of IR system is based on the Boolean model, however, the vector space model is the most frequently used in experimental systems (Shaw, Burgin, & Howell, 1997).…”

Section: Introductionmentioning

confidence: 99%

A weight-based approach to information retrieval and relevance feedback

Liao

2008

Expert Systems with Applications

View full text Add to dashboard Cite

Performance standards and evaluations in IR test collections: Vector-space and other retrieval models

Cited by 30 publications

References 32 publications

Hypergeometric language models for republished article finding

Hypergeometric language models for republished article finding

When information retrieval measures agree about the relative quality of document rankings

A weight-based approach to information retrieval and relevance feedback

Contact Info

Product

Resources

About