A probabilistic learning approach for document indexing

Abstract

We describe a method for probabilistic document indexing using relevance feedback data that has been collected from a set of queries. Our approach is based on three new concepts: (1) Abstraction from specific terms and documents, which overcomes the restriction of limited relevance information for parameter estimation.(2) Flexibility of the representation, which allows the integration of new text analysis and knowledge-based methods in our approach as well as the consideration of document structures or different types of terms.(3) Probabilistic learning or classification methods for the estimation of the indexing weights making better use of the available relevance information, Our approach can be applied under restrictions that hold for real applications.We give experimental results for five test collections which show improvements over other methods.

Text and Hypertext Categorization

“…A wide variety of learning approaches have been applied to TC, to name a few, Bayesian classification (Lewis and Ringuette 1994;Domingo and Pazzani 1996;Larkey and Croft 1996;Koller and Sahami 1997;Lewis 1998), decision trees (Weiss, Apte et al ;Fuhr and Buckley 1991;Cohen and Hirsh 1998;Li and Jain 1998), decision rule classifiers such as CHARADE (Moulinier and Ganascia 1996), or DL-ESC (Li and Yamanishi 1999), or RIPPER (Cohen and Hirsh 1998), or SCAR (Moulinier, Raskinis et al 1996), or SCAP-1 (Apté, Damerau et al 1994), multi-linear regression models (Yang and Chute 1994;Yang and Liu 1999), Rocchio method (Hull 1994;Ittner, Lewis et al 1995;Sable and Hatzivassiloglou 2000), Neural Networks (Schütze, Hull et al 1995;Wiener, Pedersen et al 1995;Dagan, Karov et al 1997;Ng, Goh et al 1997;Lam and Lee 1999;Ruiz and Srinivasan 1999), example based classifiers (Creecy 1991;Masand, Linoff et al 1992;Larkey 1999), support vector machines (Joachims 1998), Bayesian inference networks (Tzeras and Hartmann 1993;Wai and Fan 1997;Dumais, Platt et al 1998), genetic algorithms (Masand 1994;Clack, Farringdon et al 1997), and maximum entropy modelling (Manning and Schütze 1999).…”

“…combining many words as one index, for example "artificial intelligence" or "data mining" (Fuhr and Buckley 1991;Tzeras and Hartmann 1993;Schütze, Hull et al 1995). These indexes can be generated either manually or automatically.…”

“…Among these are the DIA association factor (Fuhr and Buckley 1991), chi-square (Yang and Pedersen 1997;Sebastiani, Sperduti et al 2000;Caropreso, Matwin et al 2001), NGL coefficient (Ng, Goh et al 1997;Ruiz and Srinivasan 1999), information gain Lewis and Ringuette 1994;Moulinier, Raskinis et al 1996;Yang and Pedersen 1997;Larkey 1998;Mladenic and Grobelnik 1998;Caropreso, Matwin et al 2001), mutual information (Larkey and Croft 1996;Wai and Fan 1997;Dumais, Platt et al 1998;Taira and Haruno 1999) odds ratio (Mladenic and Grobelnik 1998;Ruiz and Srinivasan 1999;Caropreso, Matwin et al 2001), relevancy score (Wiener, Pedersen et al 1995) and GSS coefficient (Galavotti, Sebastiani et al 2000). Three of the most popular methods are descrivbed briefly below.…”

A Hybrid Statistical Data Pre-processing Approach for Language-Independent Text Classification

“…• DIAAF: The Darmstadt Indexing Approach (DIA) [11] was originally "developed for automatic indexing with a prescribed indexing vocabulary" [12]. In a machine learning context, Sebastiani [23] argues that this approach "considers properties (of terms, documents, categories, or pairwise relationships among these) as basic dimensions of the learning space".…”

Source code indexing for automated tracing

“…Document indexing is defined as the task of assigning terms to documents for retrieval purposes [11]. The process consists of two generic steps: extracting the subject matter of a document, and expressing the subject matter in index terms to facilitate subject retrieval [12].…”