Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval

Robertson, Stephen; Walker, Steve

doi:10.1007/978-1-4471-2099-5_24

Cited by 769 publications

(672 citation statements)

References 6 publications

Supporting

Mentioning

633

Contrasting

Unclassified

Order By: Relevance

“…Keywords are likely to be frequent (at least inside the clusters), and documents containing many of those terms will be promoted in the rank list by the prior. This goes accordingly to the scope hypothesis [9]: documents covering many topics are more likely to be relevant.…”

Section: Probabilistic Priormentioning

confidence: 98%

“…Then, our first document length based prior is proportional to document length. The intuition behind this prior is that longer documents span more topics and are more likely to be relevant if no query has been seen (denoted as scope hypothesis in [9]). It has been reported that this prior increases the retrieval performance [6] on the WT10G collection up to 0.03 on an absolute scale.…”

Section: Linear Priormentioning

confidence: 99%

See 1 more Smart Citation

Probabilistic Document Length Priors for Language Models

Blanco

Barreiro

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. This paper addresses the issue of devising a new document prior for the language modeling (LM) approach for Information Retrieval. The prior is based on term statistics, derived in a probabilistic fashion and portrays a novel way of considering document length. Furthermore, we developed a new way of combining document length priors with the query likelihood estimation based on the risk of accepting the latter as a score. This prior has been combined with a document retrieval language model that uses Jelinek-Mercer (JM), a smoothing technique which does not take into account document length. The combination of the prior boosts the retrieval performance, so that it outperforms a LM with a document length dependent smoothing component (Dirichlet prior) and other state of the art high-performing scoring function (BM25). Improvements are significant, robust across different collections and query sizes.

show abstract

Section: Probabilistic Priormentioning

confidence: 98%

Section: Linear Priormentioning

confidence: 99%

Probabilistic Document Length Priors for Language Models

Blanco

Barreiro

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Effective information retrieval models generally capture three heuristics, i.e., TF weighting, IDF weighting, and document length normalization [36]. One effective way to assign weights to terms when representing a document as a weighted term vector is the BM25 term weighting method [78], where the normalized TF not only addresses length normalization, but also has an upper bound which improves the robustness as it avoids overly rewarding the matching of any particular term. A document can also be represented with a probability distribution over words (i.e., unigram language models), and the similarity can then be measured based an information theoretic measure such as cross entropy or Kullback-Leibler divergencce [105].…”

Section: Distance-based Clustering Algorithmsmentioning

confidence: 99%

A Survey of Text Clustering Algorithms

2012

View full text Add to dashboard Cite

Clustering is a widely studied data mining problem in the text domains. The problem finds numerous applications in customer segmentation, classification, collaborative filtering, visualization, document organization, and indexing. In this chapter, we will provide a detailed survey of the problem of text clustering. We will study the key challenges of the clustering problem, as it applies to the text domain. We will discuss the key methods used for text clustering, and their relative advantages. We will also discuss a number of recent advances in the area in the context of social network and linked data.

show abstract

“…The comparisons is between the BM25 weighting scheme (OKAPI) [12][13] and our approach. We can note from figure above (figure 3) that precision points are better for our approach for any points of precision.…”

Section: Comparisonmentioning

confidence: 99%

“…For an efficient Information Retrieval System (IRS) these two sets must be equal as often as possible. The relevance of a document to a query is usually interpreted by most of IR models, vector space [14], probabilistic [12][13] [18], inference and belief networks [20][11] [17], as a score computed by summing the inner products of term weights in the documents and query representations. Whatever the used model, the response to a user need is a list of documents ranked according to a relevance value.…”

Section: Introductionmentioning

confidence: 99%

A Model for Information Retrieval Based on Possibilistic Networks

Brini¹,

Dubois²

2005

String Processing and Information Retrieval

View full text Add to dashboard Cite

Abstract. This paper proposes a model for Information Retrieval (IR) based on possibilistic directed networks. Relations documents-terms and query-terms are modeled through possibility and necessity measures rather than a probability measure. The relevance value for the document given the query is measured by two degrees: the necessity and the possibility. More precisely, the user's query triggers a propagation process to retrieve necessarily or at least possibly relevant documents. The possibility degree is convenient to filter documents out from the response (retrieved documents) and the necessity degree is useful for document relevance confirmation. Separating these notions may account for the imprecision pervading the retrieval process. Moreover, an improved weighting of terms in a query not present in the document is introduced. Experiments carried out on a sub-collection of CLEF, namely LeMonde 1994, a French newspapers collection, showed the effectiveness of the model.

show abstract

Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval

Cited by 769 publications

References 6 publications

Probabilistic Document Length Priors for Language Models

Probabilistic Document Length Priors for Language Models

A Survey of Text Clustering Algorithms

A Model for Information Retrieval Based on Possibilistic Networks

Contact Info

Product

Resources

About