A Comprehensive Study of Features and Algorithms for URL-Based Topic Classification

Abstract

Given only the URL of a Web page, can we identify its topic? We study this problem in detail by exploring a large number of different feature sets and algorithms on several datasets. We also show that the inherent overlap between topics and the sparsity of the information in URLs makes this a very challenging problem. Web page classification without a page’s content is desirable when the content is not available at all, when a classification is needed before obtaining the content, or when classification speed is of utmost importance. For our experiments we used five different corpora comprising a total of about 3 million (URL, classification) pairs. We evaluated several techniques for feature generation and classification algorithms. The individual binary classifiers were then combined via boosting into metabinary classifiers. We achieve typical F-measure values between 80 and 85, and a typical precision of around 86. The precision can be pushed further over 90 while maintaining a typical level of recall between 30 and 40.

Supervised Term Weighting Methods for Url Classification

“…URL classification problem is studied by many researchers (Kan, 2004;Kan and Thi, 2005;Baykan et al, 2011;Rajalakshmi and Aravindan, 2011;Singh et al, 2012) and various URL features are suggested in the Science Publications JCS literature. Kan and Thi (2005) suggested segmentation techniques for extracting features from URLs.…”

“…Kan and Thi (2005) suggested segmentation techniques for extracting features from URLs. Token based features are suggested in Baykan et al (2011);Rajalakshmi and Aravindan (2011). The n-gram based approach for URL classification is discussed in (Jianping et al, 2006;Baykan et al, 2011;Rajalakshmi and Aravindan, 2013).…”

“…Token based features are suggested in Baykan et al (2011);Rajalakshmi and Aravindan (2011). The n-gram based approach for URL classification is discussed in (Jianping et al, 2006;Baykan et al, 2011;Rajalakshmi and Aravindan, 2013). In the method suggested by Jianping et al (2006), sequences of n-grams are derived from URLs and feature selection methods are applied to reduce the dimension of feature space.…”

“…3-grams alone are extracted from URLs and used as features in a fixed feature dimensional space of 26 3 in the approach suggested by Rajalakshmi and Aravindan (2013). Baykan et al (2011) derived all-gram (n-grams with n = 4 to 8) features from URLs and they used traditional tf * idf scoring method as the feature weighting method.…”

“…Baykan et al (2011) followed this weighting method for URL classification also. But for categorization tasks, training data is available with class labels and this rich source of information can be utilized in weighting the features.…”