An architecture for probabilistic concept-based information retrieval

Abstract

While concept-based methods for information retrievul can provide improved performance over more conventional techniques, they require large amounts of effort to acquire the concepts and their qualitative and quantitative relationships. This paper discusses an aschitecture for probabilistic concept-based information retrieval which addzesses the knowledge acquisition problem. The architecture makes use of the probabilistic networks technology for representing and reasoning about concepts and ineludes a knowledge acquisition component which partia]]y automates the construction of concept know1-edge bases from data.We describe two experiments that apply the architecture to the task of retrieving documents about tenorism from a set of documents from the Reuters news serv/ce. The experiments provide positive evidence that the architecture design is feasible and that there are advantages to concept-based methods. I IntroductionIn this paper we describe some preliminary research on the use of probabilistic networks for information retrieval. In particular, we introduce an architecture • This work was funded by ADS' Internal Research sad Development Prrogran~ for probabilistic, concept-based information retrieval (henceforth PCIR) that can be used to automatically generate relationships between concepts, and then reason about them given the evidence provided by individual documents. This is in contrast to our previons research on concept-based methods [12, 16,17], in which we were primarily interested in issues related to ~easonin8 about concepts and not so much concerned with issues related to the construction of the relationships. Our lons-term goal, however, ¢ontin-ues to be the development of techniques that can form the basis for an effective system to assist users in sortin 8 throu8h lurge volumes of time sensitive material. We have in mind such applications as the day-to-day momtorin 8 of newswires for specific topics of interest.

Application of Bayesian Framework in Natural Language Understanding

“…Fung [55] has developed architecture for probabilistic concept based information retrieval, addressing the problem of knowledge acquisition. Das-Newas [56] developed a belief network which allows not only for the specification of document features important to the user need, but also allows for a neutral interpretation.…”

Capturing term dependencies using a language model based on sentence trees

“…More recently, Fung and Crawford [6] have worked on concept based information retrieval that captures dependencies between 'concepts' using a Bayesian inference network. One drawback of this approach is that the user has to identify the concepts manually in each document.…”

“…If we imagine a graph of the sentence with terms as vertices and conditional probabilities from each vertex to all vertices such that as weighted, undirected edges, it is easy to see that the dependence relations in equation 7 represents a spanning tree 1 of the graph where each edge of the tree is chosen according to the relation in equation 6. The observation that the dependence relations together form a spanning tree follows from the fact that ½ A connected acyclic sub-graph spanning all vertices.…”

“…Notwithstanding this fact, it is noted that the SenTree model is still encouraging and it is hoped that the best performance can be improved by a mixture of both the models. The performance of the system that uses these values is shown in comparison to the same baseline in figure 6.…”

High-order pattern discovery from discrete-valued data

“…It finds the Markov boundary of each node (attribute, factor) in the networks which will "shield" the node from being affected by other nodes outside the boundary. CONSTRUCTOR is reported to work well when tested with training sets generated from probabilistic models and with real data in information retrieval application [17]. When going to high-order cases, the contingency table introduces a heavy computation load.…”