An integrated method for real time and offline web robot detection

Abstract

Recent academic and industry reports confirm that web robots dominate the traffic seen by web servers across the Internet. Because web robots crawl in an unregulated fashion, they may threaten the privacy, function, performance, and security of web servers. There is therefore a growing need to be able to identify robot visitors automatically, in offline and in real time, to assess their impact and to potentially protect web servers from abusive bots. Yet contemporary detection approaches, which rely on syntactic log analysis, finding statistical variations between robot and human traffic, analytical learning techniques, or complex software modifications may not be realistic to implement or remain effective as the behavior of robots evolve over time. Instead, this paper presents a novel detection approach that relies on the differences in the resource request patterns of web robots and humans. It rationalizes why differences in resource request patterns are expected to remain intrinsic to robots and humans despite the continuous evolution of their traffic. The performance of the approach, adoptable for both offline and real time settings with a simple implementation, is demonstrated by playing back streams of actual web traffic with varying session lengths and proportions of robot requests.

Efficiency Analysis Of Resource Request Patterns In Classification Of Web Robots And Humans

“…Each resource may be assigned to some resource type depending on the file extension in URI. We followed the partition of resource types proposed by Doran and Gokhale (2016), except the type corresponding to requests for directory contents, because such requests are not typical for e-commerce sites. Eight resource types were distinguished:…”

“…In (Doran and Gokhale 2011) four types of bot recognition approaches were distinguished taking into consideration the information used and techniques applied: syntactic log analysis, traffic pattern analysis, analytical learning techniques, and Turing test systems. In general, one can distinguish bot detection methods operating offline (Doran and Gokhale 2016;Lee et al 2009;Saputra et al 2013;Stassopoulou and Dikaiakos 2009;Stevanovic et al 2011;Suchacka and Sobków 2015) and online (Balla et al 2011;Doran and Gokhale 2016). In this paper we focus on offline bot detection at a Web server, when a decision on session classification (bot or human) is made given a description of the whole session (as a sequence of HTTP requests).…”

“…This is useful to develop online bot detection methods. In this study we implemented and tested the offline bot classification method proposed in (Doran and Gokhale 2016). This approach classifies bot and human sessions on a Web server based on the information on resource request patterns in session.…”

“…In contrast to most of other classification methods, this approach is conceptually simple and relatively easy to implement (it does not involve a time-consuming learning phase to infer traffic patterns). Doran and Gokhale (2016) argued that their approach is effective because resource request patterns of robots and humans constitute an intrinsic distinction between these two types of sessions which is not expected to change over time. They evaluated the efficiency of their approach using log data from three various Web servers and compared it against some other analytical learning classifiers, achieving very good classification results of their approach in terms of recall, precision, and F1.…”

“…The motivation for our experimental analysis of the approach proposed by Doran and Gokhale (2016) were some shortcomings of their experimental study. First, they used relatively old log data, which dated from 2008 to 2011, depending on a dataset.…”