Analysis of Web Spam for Non-English Content: Toward More Effective Language-Based Classifiers

Abstract

Web spammers aim to obtain higher ranks for their web pages by including spam contents that deceive search engines in order to include their pages in search results even when they are not related to the search terms. Search engines continue to develop new web spam detection mechanisms, but spammers also aim to improve their tools to evade detection. In this study, we first explore the effect of the page language on spam detection features and we demonstrate how the best set of detection features varies according to the page language. We also study the performance of Google Penguin, a newly developed anti-web spamming technique for their search engine. Using spam pages in Arabic as a case study, we show that unlike similar English pages, Google anti-spamming techniques are ineffective against a high proportion of Arabic spam pages. We then explore multiple detection features for spam pages to identify an appropriate set of features that yields a high detection accuracy compared with the integrated Google Penguin technique. In order to build and evaluate our classifier, as well as to help researchers to conduct consistent measurement studies, we collected and manually labeled a corpus of Arabic web pages, including both benign and spam pages. Furthermore, we developed a browser plug-in that utilizes our classifier to warn users about spam pages after clicking on a URL and by filtering out search engine results. Using Google Penguin as a benchmark, we provide an illustrative example to show that language-based web spam classifiers are more effective for capturing spam contents.

Arabic Natural Language Processing and Machine Learning-Based Systems

“…However, as the number of new spam types increases every year, this field should be up to date to eliminate the negative impacts of these new types of spam. By considering the features of a web page language, spam detection tools would be highly effective at capturing spam content [25].…”

“…Web spamming can be very dangerous because it spreads malware, which can affect users' privacy by obtaining sensitive information. Web spam detection using ANLP tools are used to determine information and features included in the content of web pages to ensure that only web pages that present useful content are retrieved [18], [25]- [28]. This allows threats posed by suspicious web pages to be mitigated [3].…”

“…3) DECISION TREES have been used in many NLP-related classification problems [61]. In addition to the Naive Bayes classifier, Decision Tree classifiers provide excellent results for spam detection [18], [25], [27], [28]. The decision tree classifier is a favored machine learning technique because the model is easy to understand.…”

Detecting Malicious URLs Using Machine Learning Techniques: Review and Research Directions

“…When URLs are exploited for purposes other than accessing legitimate resources on the Internet, they pose a threat to data integrity, confidentiality, and availability. The different kinds of malicious URLs are discussed below [9].…”

Performance-Based Comparative Assessment of Open Source Web Vulnerability Scanners

“…Prior work addressed web security vulnerabilities by either researching the ways to improve dynamic testing techniques [41], proposing or evaluating static analysis testing mechanisms [42][43][44][45], or using these testing approaches to evaluate the security of web-based systems [46][47][48][49]. The majority of research studies that evaluated the performance of web vulnerability scanners suggested that they have limited crawling capabilities and high false positive rates.…”