Combining compound and single terms under language model framework

Abstract

Most existing Information Retrieval model including probabilistic and vector space models are based on the term independence hypothesis. To go beyond this assumption and thereby capture the semantics of document and query more accurately, several works have incorporated phrases or other syntactic information in IR, such attempts have shown slight benefit, at best. Particularly in language modeling approaches this extension is achieved through the use of the bigram or n-gram models. However, in these models all bigrams/ngrams are considered and weighted uniformly. In this paper we introduce a new approach to select and weight relevant n-grams associated with a document. Experimental results on three TREC test collections showed an improvement over three strongest state-of-the-art model baselines, which are the original unigram language model, the Markov Random Field model, and the positional language model.

Enhancing information retrieval through concept‐based language modeling and semantic smoothing

“…We think that it can be further enhanced by considering frequent subconcepts as the approach proposed in Hammache et al. (). Our approach of recognizing concepts is somehow limited by order because we extract first the collocations that vary from one to three words. We can overcome this limitation by relaxing concept word adjacency and recognizing concepts in a larger context such as a text passage. The number of trials for this experiment are the total number of queries in the WSJ data set (49) plus the total number from the AP data set (47).…”

“…() and Hammache et al. (), in the same way, estimate concept weights. The intuition is that the authors tend to use subconcepts to refer to a given concept they have previously used in the document.…”

“…Along these lines, Hammache et al. () combined single words and filtered bigrams into a language model. They proposed an approach for filtering bigrams and weighting them by considering both their frequencies and their component terms in the document collection.…”

“…More particularly, in the context of semantic IR, a wide range of LM approaches have been proposed to relax the word independency assumption. Most approaches have focused on considering term dependencies such as n‐grams, phrases, co‐occurrence, or ontological concepts (Cao et al., ; Gao, Nie, Wu, & Cao, ; Hammache, Boughanem, & Ahmed‐Ouamar, ; Srikanth & Srihari, ; Zhou, Hu, & Zhang, ) in an attempt to capture user query and document semantics. Two main types of approaches can be distinguished, namely, corpus‐based approaches and external resource‐based approaches .…”