Feature subset selection in text-learning

Abstract

Abstract. This paper describes several known and some new methods for feature subset selection on large text data. Experimental comparison given on real-world data collected from Web users shows that characteristics of the problem domain and machine learning algorithm should be considered when feature scoring measure is selected. Our problem domain consists of hyperlinks given in a form of small-documents represented with word vectors. In our learning experiments naive Bayesian classifier was used on text data. The best performance was achieved by the feature selection methods based on the feature scoring measure called Odds ratio that is known from information retrieval. I n t r o d u c t i o nIn propositional learning problem domain is given by a set of examples, where each example is described with a class value and a vector of feature values. Features used to describe examples are not necessary all relevant and beneficial for the inductive learning and may reduce quality of induced model. Additionally, a high number of features may slow down the induction process while giving similar results as obtained with much smaller feature subset. Section 2 describes approach commonly used for feature subset selection in learning on text data (text-learning). In Section 4 we experimentally compare different feature scoring measures on real-world data collected from Web users. Section 3 describes our problem domain and naive Bayesian classifier for text that we used in experiments. Discussion is given in Section 5. 2F e a t u r e s u b s e t s e l e c t i o n approaches Different methods have been developed and used for feature subset selection in statistics, p~ttern recognition and machine learning, using different search strategies and evaluation functions. John et al. [4] pointed out the difference between the two main approaches used in machine learning to feature subset selection: filtering approach where the feature subset is selected independent of the learning method and wrapper approach where the feature subset is selected using the same learning algorithm that will be used for learning on domain represented with the selected feature subset.

Mining HTML Pages to Support Document Sharing in a Cooperative System

“…The combination information gain and Bayesian classifier outperform all methods only in the experiment on 2,000 training pages. Our results are at divergence with those reported in the work by Mladenic [12], according to which information gain performs worse than random feature selection. A closer look at words sorted by information gain shows that almost all best words are characteristic of a negative class value.…”

“…For two-class problems, Mladenic [12] compared scoring measures based on the Odds ratio 1 and those based on information gain, 2 leading her to favor the former. For multi-class problems, as in the case of WebClass, an extension of the well-known TF-IDF measure to text categorization, originally proposed for information retrieval purposes [17], has been suggested [6].…”

Protecting Sensitive Topics in Text Documents with PROTEXTOR

“…For each list, suppress words from x until the log-likelihood (LL) of y is less than the LL of the k closest classes. In addition to the PosMin metric as described in Sec 2, we used three standard feature selection metrics described in [4]: InfoGain, OddsRatio, and FreqLogP. Using a leave-one-out procedure we trained new naive Bayes and SVM classifiers on the corpus minus each redacted document, and tested on the redacted sets.…”

Classifying Structured Web Sources Using Aggressive Feature Selection

“…All performance values reported were obtained using 2-fold cross-validation scheme. In addition to our two metrics, we carried out experiments with other reportedly most effective metrics: χ 2 (CHI), the multi-class version of Information Gain (IG), Document Frequency (DF) (Yang and Pedersen, 1997), the binary version of Information Gain (IG2) (Gabrilovich and Markovitch, 2004), Bi-normal separation (BNS) (Forman, 2003), Odds Ratio (OR) (Mladenic, 1998). For metrics with one value per category (χ 2 , IG2, BNS, OR), we used the maximum value as the score, for it performs better than the average value across metrics, classifiers, and text collections (Rogati and Yang, 2002).…”

“…Third, the problem of feature selection in text categorization has been intensively studied, e.g., in (Yang and Pedersen, 1997;Mladenic, 1998;Soucy and Mineau, 2001;Forman, 2003;Gabrilovich and Markovitch, 2004). These works, as well as our methods, use the filtering approach, in which terms are scored by a metric, then the highest ranked terms are selected (Sebastiani, 2002).…”