Two-Stage Document Length Normalization for Information Retrieval

Abstract

The standard approach for term frequency normalization is based only on the document length. However, it does not distinguish the verbosity from the scope, these being the two main factors determining the document length. Because the verbosity and scope have largely different effects on the increase in term frequency, the standard approach can easily suffer from insufficient or excessive penalization depending on the specific type of long document. To overcome these problems, this paper proposes two-stage normalization by performing verbosity and scope normalization separately, and by employing different penalization functions. In verbosity normalization, each document is pre-normalized by dividing the term frequency by the verbosity of the document. In scope normalization, an existing retrieval model is applied in a straightforward manner to the pre-normalized document, finally leading us to formulate our proposed verbosity normalized (VN) retrieval model. Experimental results carried out on standard TREC collections demonstrate that the VN model leads to marginal but statistically significant improvements over standard retrieval models.

A Pólya Urn Document Language Model for Improved Information Retrieval

“…Interestingly, recent research has developed a two-stage document length normalisation framework [Na 2015] which incorporates both verbosity and scope normalisation into retrieval methods. It is appealing that the SPUD retrieval methods derived from our probabilistic framework contain these aspects of normalisation naturally.…”

An Axiomatic Approach to Diagnosing Neural IR Models

“…Fang et al [14] also proposed the use of perturbed document collections to gather further insights on retrieval functions fulfilling the same set of axioms. This approach has not been followed-up upon in works other than [31].…”

A retrieval model family based on the probability ranking principle for ad hoc retrieval

“…where k, k 0 , and b are constants and Δ is the average document length. The TF component of the BM25 ranking function incorporates document length normalization, which ensures long documents are not excessively favored over short documents in retrieval (Na, 2015;Singhal et al, 1996). Instead of a simple normalization by the document length j d j, the normalization in BM25 takes into account that the length of a document may depend on the document's verbosity and scope (Robertson & Walker, 1994).…”

“…A document d may be represented as a vector with each dimension and its value corresponding to a term t in d and the TF, respectively. The use of TF normalized by the document length in ranking functions can enhance retrieval effectiveness (Na, 2015;Singhal et al, 1996). We denote the normalized TF by f t, d ð Þ and represent the document with normalized TF, d, by a set of tuples, that is,…”