An Automatic Scheme to Categorize User Sessions in Modern HTTP Traffic

Lin, Xiao-zhu; Quan, Lin; Wu, Haiyan

doi:10.1109/glocom.2008.ecp.290

Cited by 10 publications

(9 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Early detection techniques are based on syntactical log analysis (Kabe & Miyazaki, ; Community, ) and continue to serve as a useful way for identifying web robots (Huntington et al, ) that are known and have recognizable ip addresses and user‐agent strings. Traffic pattern analysis: Traffic‐based analysis techniques search for statistical contrasts between the characteristics of robot and human traffic. The methods find contrasts according to fixed expectations about robot and human behaviors (Jansen, Spink, & Saracevic, ; Guo et al, ; Geens et al, ; Lin, Quan, & Wu, ; Duskin & Feitelson, ; Hayati, Potdar, Talevski, & Smyth, ; Kwon, Kim, & Cha, ; Kwon et al, ; Bai, Xiong, Zhao, & He, ). For example, a traffic analysis technique may check how similar a session's navigational pattern is to a depth‐first or breadth‐first search of the hyperlinks of a site ‐ a pattern that an analyst may assume robot sessions would exhibit. Analytical learning techniques: Analytical learning techniques exploit the observed characteristics of the logged sessions to estimate the likelihood that a given session was generated by a robot with a machine learning algorithm (Doran & Gokhale, ).…”

Section: Perspective On Web Robot Detectionmentioning

confidence: 99%

“…• Traffic pattern analysis: Traffic-based analysis techniques search for statistical contrasts between the characteristics of robot and human traffic. The methods find contrasts according to fixed expectations about robot and human behaviors (Jansen, Spink, & Saracevic, 2000;Guo et al, 2005;Geens et al, 2006;Lin, Quan, & Wu, 2008;Duskin & Feitelson, 2009;Hayati, Potdar, Talevski, & Smyth, 2010;Kwon, Kim, & Cha, 2012a;Kwon et al, 2012b;Bai, Xiong, Zhao, & He, 2014). For example, a traffic analysis technique may check how similar a session's navigational pattern is to a depth-first or breadth-first search of the hyperlinks of a site -a pattern that an analyst may assume robot sessions would exhibit.…”

Section: Perspective On Web Robot Detectionmentioning

confidence: 99%

“…Traffic pattern analysis Discover statistical differences in request-or session-level features (Geens et al, 2006), (Guo et al, 2005), (Duskin & Feitelson, 2009), (Jansen et al, 2000), (Lin et al, 2008), (Kwon et al, 2012b), (Kwon et al, 2012a), , (Bai et al, 2014) Patterns studied are based on fixed expectation about robot behaviors; expectations may change over time…”

Section: Present Limitationsmentioning

confidence: 99%

See 2 more Smart Citations

An integrated method for real time and offline web robot detection

Gokhale

2016

Expert Systems

View full text Add to dashboard Cite

Recent academic and industry reports confirm that web robots dominate the traffic seen by web servers across the Internet. Because web robots crawl in an unregulated fashion, they may threaten the privacy, function, performance, and security of web servers. There is therefore a growing need to be able to identify robot visitors automatically, in offline and in real time, to assess their impact and to potentially protect web servers from abusive bots. Yet contemporary detection approaches, which rely on syntactic log analysis, finding statistical variations between robot and human traffic, analytical learning techniques, or complex software modifications may not be realistic to implement or remain effective as the behavior of robots evolve over time. Instead, this paper presents a novel detection approach that relies on the differences in the resource request patterns of web robots and humans. It rationalizes why differences in resource request patterns are expected to remain intrinsic to robots and humans despite the continuous evolution of their traffic. The performance of the approach, adoptable for both offline and real time settings with a simple implementation, is demonstrated by playing back streams of actual web traffic with varying session lengths and proportions of robot requests.

show abstract

Section: Perspective On Web Robot Detectionmentioning

confidence: 99%

Section: Perspective On Web Robot Detectionmentioning

confidence: 99%

Section: Present Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

An integrated method for real time and offline web robot detection

Gokhale

2016

Expert Systems

View full text Add to dashboard Cite

show abstract

“…Lin et al introduce a scheme to categorize user sessions into different groups (Lin et al 2008). This study postulates that modern Web traffic is multi-class, consisting of humans, Web robots, and other Internet protocols, such as peer-to-peer file sharing.…”

Section: Detection Using Traffic Metricsmentioning

confidence: 99%

Web robot detection techniques: overview and limitations

Doran

Gokhale

2010

Data Min Knowl Disc

View full text Add to dashboard Cite

Most modern Web robots that crawl the Internet to support value-added services and technologies possess sophisticated data collection and analysis capabilities. Some of these robots, however, may be ill-behaved or malicious, and hence, may impose a significant strain on a Web server. It is thus necessary to detect Web robots in order to block undesirable ones from accessing the server. Such detection is also essential to ensure that the robot traffic is considered appropriately in the performance and capacity planning of Web servers. Despite a variety of Web robot detection techniques, there is no consensus regarding a single technique, or even a specific "type" of technique, that performs well in practice. Therefore, to aid in the development of a practically applicable robot detection technique, this survey presents a critical analysis and comparison of the prevalent detection approaches. We propose a framework to classify the existing detection techniques into four categories based on their underlying detection philosophy. We compare the different classes to gain insights into those characteristics that make up an effective robot detection scheme. Finally, we discuss why the contemporary techniques fail to offer a general solution to the robot detection problem and propose a set of key ingredients necessary for strong Web robot detection.

show abstract

“…Other studies [6, 11] have considered the ranking of web pages by popularity. Lee et al [11] developed an effective characterization metric, based on workload characteristics and resource types, in detecting and classifying various web robots. They proposed that the popularity of web pages referenced by human beings is highly concentrated.…”

Section: Related Workmentioning

confidence: 99%

Web robot detection based on pattern-matching technique

Kwon

Kim

Cha

2012

Journal of Information Science

View full text Add to dashboard Cite

In web robot detection it is important is to find features that are common characteristics of diverse robots, in order to differentiate between them and humans. Existing approaches employ fairly simple features (e.g. empty referrer field, interval between successive requests), which often fail to reflect web robots' behaviour accurately. False alarms may therefore occur unacceptably often. In this paper we propose a fresh approach that expresses the behaviour of interactive users and various web robots in terms of a sequence of request types, called request patterns. Previous proposals have primarily targeted the detection of text crawlers, but our approach works well on many other web robots, such as image crawlers, email collectors and link checkers. In empirical evaluation of more than 1 billion requests collected at www.microsoft.com, our approach achieved 94% accuracy in web robot detection, estimated by F-measure. A decision tree algorithm proposed by Tan and Kumar was also applied to the same data. A comparison shows that the proposed approach is more accurate, and that real-time detection of web robots is feasible.

show abstract

An Automatic Scheme to Categorize User Sessions in Modern HTTP Traffic

Cited by 10 publications

References 15 publications

An integrated method for real time and offline web robot detection

An integrated method for real time and offline web robot detection

Web robot detection techniques: overview and limitations

Web robot detection based on pattern-matching technique

Contact Info

Product

Resources

About