An industrial perspective on web scraping characteristics and open issues

Abstract

An ongoing battle has been running for more than a decade between e-commerce websites owners and web scrapers. Whenever one party finds a new technique to prevail, the other one comes up with a solution to defeat it. Based on our industrial experience, we know this problem is far from being solved. New solutions are needed to address automated threats. In this work, we will describe the actors taking part in the battle, the weapons at their disposal, and their allies on either side. We will present a real-world setup to explain how e-commerce websites operators try to defend themselves and the open problems they seek solutions for.

BADPASS: Bots Taking ADvantage of Proxy as a Service

“…In [27], Li et al show that more than half of the bot ip addresses they collected with their honeypots belong to residential networks. As explained in [16], current detection techniques struggle in blocking bots using such ips.…”

“…Nowadays, websites in different domains, such as e-commerce, ticketing, and social media, are engaged in a persistent fight against subtle but damaging actors: scraping bots. They produce a significant amount of traffic towards these websites producing large financial losses, as explained in recent works [23,16].…”

ImMuNE: Improved Multilateration in Noisy Environments

“…In [1], Chiapponi et al present the impact of residential IP proxies on web scraping campaigns. In [5], the same authors use the semantics of the received queries to group searches issued by different IP addresses, leading to the conclusion that Client Server SUPERPROXY ...…”

“…Scraping bots are a plague for online companies. They continuously query websites, increasing the costs for their owners without generating any revenue [1]. Commercial antibot solutions exist to counter this threat.…”