Towards a framework for detecting advanced Web bots

Abstract

Automated programs (bots) are responsible for a large percentage of website traffic. These bots can either be used for benign purposes, such as Web indexing, Website monitoring (validation of hyperlinks and HTML code), feed fetching Web content and data extraction for commercial use or for malicious ones, including, but not limited to, content scraping, vulnerability scanning, account takeover, distributed denial of service attacks, marketing fraud, carding and spam. To ensure their security, Web servers try to identify bot sessions and apply special rules to them, such as throttling their requests or delivering different content. The methods currently used for the identification of bots are based either purely on rule-based bot detection techniques or a combination of rulebased and machine learning techniques. While current research has developed highly adequate methods for Web bot detection, these methods' adequacy when faced with Web bots that try to remain undetected hasn't been studied. For this reason, we created and evaluated a Web bot detection framework on its ability to detect conspicuous bots separately from its ability to detect advanced Web bots. We assessed the proposed framework performance using real HTTP traffic from a public Web server. Our experimental results show that the proposed framework has significant ability to detect Web bots that do not try to hide their bot identity using HTTP Web logs (balanced accuracy in a false-positive intolerant server > 95%). However, detecting advanced Web bots that present a browser fingerprint and may present a humanlike behaviour as well is considerably more difficult.

Online web navigation assistant

“…The phase incorporates the following steps: Data Cleaning. Cleaning and filtering of irrelevant, redundant, and noisy data that is not suitable for the mining process [32,33].…”

Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics

“…Such web bots may though be abused for malicious purposes. These malicious web bots can perform highly complex tasks and at the same time try to avoid detection by presenting a browser fingerprint and a humanlike behaviour [14,15,20]. This makes them particularly dangerous, since they present themselves as humans and perform several actions in a humanlike way, which severely hinders their detectability.…”

“…Current research on machine learning-based web bot detection uses either web logs [3,12,13,29,34,37,38] or mouse movements [12,42] to detect web bots. Even though the aforementioned techniques are highly accurate, they do not address one key aspect of the problem, which is that web bots might try to exhibit both a browser like fingerprint and a humanlike behaviour to avoid detection [15,20].…”

“…This work uses as a basis and expands the framework proposed in Reference [20] and addresses the challenges that the authors outlined. More specifically, to overcome the limitations of the web bot detection framework in Reference [20] in detecting advanced web bots based on features extracted from web logs, this framework (i) introduces an additional module that detects web bots based on their mouse movements, and (ii) proposes a novel way of combining the detection modules utilised (i.e., the one that uses web logs and the one that uses mouse movements) to enable capturing the different temporal characteristics of the web logs and the mouse movements, as well as the spatial characteristics of the mouse movements.…”

“…Additionally, in this research, we further split the web bots that were defined as "advanced" in Reference [20] into moderate web bots and advanced web bots, both of which present a browser fingerprint but otherwise have different levels of sophistication; this differs from Reference [20] where web bots were split based on whether they present a browser fingerprint or not. This work does not consider the simple web bots (i.e., simple scripts) of Reference [20], since they have a botlike fingerprint and perform no mouse movements, hence they can be detected in a deterministic way.…”