Fast and accurate text classification via multiple linear discriminant projections

Abstract

Abstract. Support vector machines (SVMs) have shown superb performance for text classification tasks. They are accurate, robust, and quick to apply to test instances. Their only potential drawback is their training time and memory requirement. For n training instances held in memory, the best-known SVM implementations take time proportional to n a , where a is typically between 1.8 and 2.1. SVMs have been trained on data sets with several thousand instances, but Web directories today contain millions of instances that are valuable for mapping billions of Web pages into Yahoo!-like directories. We present SIMPL, a nearly linear-time classification algorithm that mimics the strengths of SVMs while avoiding the training bottleneck. It uses Fisher's linear discriminant, a classical tool from statistical pattern recognition, to project training instances to a carefully selected low-dimensional subspace before inducing a decision tree on the projected instances. SIMPL uses efficient sequential scans and sorts and is comparable in speed and memory scalability to widely used naive Bayes (NB) classifiers, but it beats NB accuracy decisively. It not only approaches and sometimes exceeds SVM accuracy, but also beats the running time of a popular SVM implementation by orders of magnitude. While describing SIMPL, we make a detailed experimental comparison of SVM-generated discriminants with Fisher's discriminants, and we also report on an analysis of the cache performance of a popular SVM implementation. Our analysis shows that SIMPL has the potential to be the method of choice for practitioners who want the accuracy of SVMs and the simplicity and speed of naive Bayes classifiers.

Context‐based term frequency assessment for text classification

“…20NG contains 19,997 documents of 20 topics (categories). As in Chakrabarti et al (2003), we employ 75% of the documents for training, and the remaining 25% for testing. Therefore, there are 14,997 training documents and 5,000 test documents, which are uniformly extracted from the original 19,997 documents.…”

“…SVM is a popular technique in TC (e.g., Bennett & Nguyen, 2009; Xue et al, 2008; Qi & Davison, 2008; Chakrabarti et al, 2003; Yang & Lin, 1999). Previous studies often found that SVM outperforms many classifiers.…”

“…Previous studies often found that SVM outperforms many classifiers. We employ SVM Light that is publicly available9 (Joachims, 1999) and was tested in many previous studies (e.g., Chakrabarti et al, 2003; Yang & Lin, 1999). We set the “cost‐factor” parameter to 10 (i.e., relative weights of errors on positive examples to errors on negative examples), which helps SVM to have a performance similar to the best performance reported in previous studies (Experimental Results, below).…”

Text Classification Algorithms: A Survey

“…Texts and documents, especially with weighted feature extraction, generate a huge number of features. Many researchers have applied random projection to text data [83,84] for text mining, text classification, and dimensionality reduction. In this section, we review some basic random projection techniques.…”

Text classification based on deep belief network and softmax regression

“…With increasing scientific papers, Internet information, and other text-format data, automatic text categorization plays an important role in information retrieval, data mining and machine learning [13]. Commonly used classification methods are back propagation neural network, decision trees, K-nearest neighbor (KNN), naive Bayes and SVM, and especially SVM achieves good performance on the effectiveness and stability of classification [14][15][16][17][18][19][20][21][22][23][24][25][26][27][28]. However, most of them are supervised learning algorithms and training data or labeled samples often demand great human efforts in practical applications.…”