CARONTE: Crawling Adversarial Resources Over Non-Trusted, High-Profile Environments

Abstract

The monitoring of underground criminal activities is often automated to maximize the data collection and to train ML models to automatically adapt data collection tools to different communities. On the other hand, sophisticated adversaries may adopt crawling-detection capabilities that may significantly jeopardize researchers' opportunities to perform the data collection, for example by putting their accounts under the spotlight and being expelled from the community. This is particularly undesirable in prominent and high-profile criminal communities where entry costs are significant (either monetarily or for example for background checking or other trust-building mechanisms). This paper presents CARONTE, a tool to semiautomatically learn virtually any forum structure for parsing and data-extraction, while maintaining a low profile for the data collection and avoiding the requirement of collecting massive datasets to maintain tool scalability. We showcase the tool against four underground forums, and compare the network traffic it generates (as seen from the adversary's position, i.e. the underground communitys server) against state-of-the-art tools for web-crawling as well as human users.

Detection of Advanced Web Bots by Combining Web Logs with Mouse Behavioural Biometrics

“…Although the aforementioned techniques show promising results, they do not address a key aspect of the web bot detection problem, which is the identification of web bots that try to evade detection via, for example, exhibiting a browser fingerprint and a humanlike behaviour [11,20,21,27]. Based on the current detection techniques, advanced web bots could be modelled by adversaries by (i) having a browser fingerprint, and (ii) exhibiting a humanlike behaviour.…”

“…A fingerprint that is close to one of a browser can be be achieved with the use and configuration of specific browsing automation software [11,27,35], or by using regular browsers [2]. A humanlike behaviour can be achieved by time sleeps that take into account the length of text in each web page [11,27], crawling web pages at a specific time (e.g., day/night) [11,27], skipping or spending a few seconds on content that they have already visited [11], and performing human assisted [27] or automatic [11] logging in to websites. Recent research has shown that, although detecting simple bots is relatively easy, detecting more sophisticated web bots that use a browser fingerprint and/or exhibit a humanlike behaviour, including the performance of mouse movements, is much more difficult [20].…”

“…As there is neither a universal definition of how the moderate and advanced web bots behave, nor are there any existing tools that we can use to generate them, we based these behaviours on the information available in literature [11,21,27], as well as in reports of web bot detection companies [15]. More specifically, we consider the moderate web bots to have no intelligence, meaning that they follow hyperlinks randomly and by performing direct mouse movements between those links.…”

“…Additionally, advanced web bots can emulate "reading" part of a web page by performing mouse movements in a left to right direction and back, as if to follow the direction of the text (like humans sometimes do). As a result, the time between requests is also adjusted based on whether they are "reading" the web page or not, and based on the content of the web page that is read [11].…”

“…The further categorisation of the web bots referred to as "advanced" in Reference [20] into "moderate" and "advanced" web bots in this work, on the basis of their evasiveness, allows us to further showcase how the increase in their evasiveness capabilities affects the performance of our detection models. The behaviour and configuration of the tested web bots is derived from techniques that have been proposed in literature for evasive web content gathering [11,21,27] in combination with techniques that stem from the observation of advanced web bots in the wild [14,15].…”