The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing

Abstract

Large-scale Web crawls have emerged as the state of the art for studying characteristics of the Web. In particular, they are a core tool for online tracking research. Web crawling is an attractive approach to data collection, as crawls can be run at relatively low infrastructure cost and don't require handling sensitive user data such as browsing histories. However, the biases introduced by using crawls as a proxy for human browsing data have not been well studied. Crawls may fail to capture the diversity of user environments, and the snapshot view of the Web presented by one-time crawls does not reflect its constantly evolving nature, which hinders reproducibility of crawl-based studies. In this paper, we quantify the repeatability and representativeness of Web crawls in terms of common tracking and fingerprinting metrics, considering both variation across crawls and divergence from human browser usage. We quantify baseline variation of simultaneous crawls, then isolate the effects of time, cloud IP address vs. residential, and operating system. This provides a foundation to assess the agreement between crawls visiting a standard list of high-traffic websites and actual browsing behaviour measured from an opt-in sample of over 50,000 users of the Firefox Web browser. Our analysis reveals differences between the treatment of stateless crawling infrastructure and generally stateful human browsing, showing, for example, that crawlers tend to experience higher rates of third-party activity than human browser users on loading pages from the same domains.

OmniCrawl: Comprehensive Measurement of Web Tracking With Real Desktop and Mobile Browsers

“…Our evaluation focuses on the scope of validity of automated crawlers, real user behavior notwithstanding. To this extent, our comparison of baseline variation is similar to Zeber et al's [107]; we also reach the conclusion that automated crawler visiting sites at the same time receive similar content and thus can serve as a basis for comparison of other variables.…”

“…Several studies investigate the representativeness of automated crawls in comparison to real user behavior [52,107] and suggest that automated crawlers might over-approximate the amount of third-party trackers a real user would experience. This is a limitation inherent to automated crawlers that may be ameliorated by future work on informing crawler behavior with real user behavior.…”

Mis-shapes, Mistakes, Misfits

“…For retrieving Web site contents through Web crawls, Ahmad et al developed a framework to compare crawlers based on varying technologies, finding that the choice of crawler may significantly impact measurements [105]. Zeber et al compared crawlers with each other and with human user traffic, and found results to vary over time as well as across platforms [106]. We provide a similar assessment of domain classification services, as they can equally impact the results of research studies.…”

FP-Redemption: Studying Browser Fingerprinting Adoption for the Sake of Web Security

“…Past studies used automated crawls to observe browser fingerprinting at scale [22,1,8]. However, relying on bot crawls introduces bias in the collected data [14,11,34] as more and more websites use defenses to block bot access [33]. Automating the registration and payment processes is also a challenge because of the high variability that can be found in related forms [13].…”