Document and Passage Retrieval Based on Hidden Markov Models

Abstract

Introduced is a new approach to Information Retrieval developed on the basis of Hidden Markov Models (HMMs). HMMs are shown to provide a mathematically sound framework for retrieving documents-documents with predefined boundaries and also entities of information that are of arbitrary lengths and formats (passage retrieval). Our retrieval model is shown to encompass promising capabilities: First, the position of occurrences of indexing features can be used for indexing. Positional information is essential, for instance, when considering phrases, negation, and the proximity of features. Second, from training collections we can derive automatically optimal weights for arbitrary features. Third, a query dependent structure can be determined for every document by segmenting the documents into passages that are either relevant or irrelevant to the query. The theoretical analysis of our retrieval model is complemented by the results of preliminary experiments. Introd uctionWe introduce a new approach to Information Retrieval, i.e. document retrieval and passage retrieval. Documents are considered as being produced by stochastic processes. A first stochastic process generates text fragments that are relevant to a certain query. A second stochastic process generates text fragments independent of any particular query. The generation of text fragments by the two stochastic processes is modeled by means of two Hidden Markov Models (HMMs). Whether one of these two HMMs generates a text fragment with a high probability or with a low probability depends on the distribution of the query features within the text fragment.In the case of document retrieval, the documents are assigned scores that depend on the ratio of the probability that the document was generated by the first stochastic process and the probability that the document was generated by the second stochastic process. As usual, the documents are presented to the user in decreasing order of their scores. In the case of passage retrieval, the score of a passage depends on the probability that the passage itself was generated by the first stochastic process and the text fragments before and after the passage are generated by the second stochastic process.There are three problems, that are considered difficult in Information Retrieval. First, it is not well known how complex features (e.g. phrases, proximity data, negations, cooccurrence and cocitation data etc.) should be used for i~dexing. Second, we lack a general weighting scheme for arbitrary indexing features and for arbitrary document collections. Third, the optimal segmentation of a long document into segments that are either relevant or irrelevant to a query is another open problem. Our approach encompasses promising capabilities to solve these three problems at least partially. First, information about positions of features can be conserved, because in our approach a document is considered as being produced by a stochastic process. Conventional retrieval models do not take into account the positions where...

UMass at TREC 2004: Novelty and HARD

“…Previous work has been inconclusive as to whether there is benefit to retrieving passages of different lengths [13,3,25]. However, most past studies have only evaluated passage retrieval by its ability to retrieve relevant documents, due in part to the unavailability of passagelevel relevance judgments.…”

Combining Approaches to Information Retrieval

“…Discourse passages are based on textual discourse units such as sentences, paragraphs and sections (e.g., Salton et al, 1993;Wilkinson, 1994). Semantic passages are based on similarities of the subject or content of the text (e.g., Hearst and Plaunt, 1993;Mittendorf and Schauble, 1994). Window passages are based upon a fixed number of words (Callan, 1994).…”

“…For example, Salton et al, 1993, refine rankings based on global similarity using sentence, paragraph and section similarities. Mittendorf and Schauble, 1994, combine a probabilistic model of relevant text passages with a model of text in general. Callan, 1994, combines global document evidence and window-based passage evidence in a probabilistic framework.…”

“…For the purposes of document retrieval, Ponte and Croft modeled occurrences at the document level without regard to sequential effects, although showed that it is possible to model local predictive effects for features such as phrases. Mittendorf and Schauble, 1994, used a similar approach to construct a generative model for retrieval based on document passages. The approach to retrieval described in Ponte and Croft, 1998, is to infer a language model for each document and to estimate the probability of generating the query according to each of these models.…”

Mining conversational text for procedures with applications in contact centers

“…Automatically assigning quality scores to calls in contact centers [4], mining call transcripts for trend analysis [5] and call-flow based analysis of call center transcripts [6] are interesting research topics in the contact center analysis. Segmentation of conversation transcripts has been attempted in the past [7] for information retrieval and summarization [8]. Noisy conversational text has been looked at for adding sentence boundaries [9].…”