Catching web crawlers in the act

Abstract

This paper recommends a new approach to the detection and containment of Web crawler traverses based on clickstream data mining. Timely detection prevents crawler abusive consumption of Web server resources and eventual site contents privacy or copyrights violation. Clickstream data differentiation ensures focused usage analysis, valuable both for regular users and crawler profiling. Our platform, named ClickTips, sustains a sitespecific, updatable detection model that tags Web crawler traverses based on incremental Web session inspection and a decision model that assesses eventual containment. The goal is to deliver a model flexible enough to keep up with crawling continuous evolving and that is capable of detecting crawler presence as soon as possible. We use a real-world Web site case study as a support for process description, as well as, to evaluate the accuracy of the obtained classification models and their ability for discovering previously unknown Web crawlers.

Classification of web robots: An empirical study based on over one billion requests

“…Our analysis reveals that such observation is generally, but not always, true. Among all the logs, unassigned referrer field was found in about 25% of the requests, which is about half of the number reported in an earlier study (Lourenco and Belo, 2006). Among the rest, about 60% contained URL external to Microsoft.…”

“…Dikaiakos and Stassopoulou Kumar, 2000, 2002) Tan and Kumer (Dikaiakos et al, 2003(Dikaiakos et al, , 2005Stassopoulou and Dikaiakos, 2006) Almeida et al (Almeida et al, 2001) Lourenco and Belo (Lourenco and Belo, 2006) They used most of the information included in the log including some variations on traffic patterns. In particular, they examined HTTP-traffic characteristics (e.g., methods, error codes, etc) as well as resource referencing behavior (e.g., file type, percentage of distinct requests, resource popularity, and concentration of requests, etc).…”

“…Unassigned referrer field is often considered one of the most apparent characteristics of web robots Kumar, 2000, 2002;Lourenco and Belo, 2006). As typical web robots parse a page and build up the list of pages to be visited, referrer field is frequently left blank.…”

Web Usage Mining for Discovery and Evaluation of Online Navigation Pattern Prediction

“…The crawler will automatic crawl information to avoid this robot.txt file will be removed [17]. In this paper, one lexical pattern is defined in which the robot.txt file will never be crawled.…”

Evaluating Web Site Structure Based on Navigation Profiles and Site Topology

“…The present work used the pattern mining approach introduced in [4,3], involving the semi-automatic labeling of a training set of Web sessions and tree model induction. Besides crawler and regular user sessions there were identified browser-related application sessions ( …”