UbiCrawler: a scalable fully distributed Web crawler

Boldi, Paolo; Codenotti, Bruno; Santini, Massimo; Vigna, Sebastiano

doi:10.1002/spe.587

Cited by 401 publications

(247 citation statements)

References 15 publications

Supporting

Mentioning

243

Contrasting

Unclassified

Order By: Relevance

“…[3] first proposed a series of basic concepts on parallel crawling and distributed crawling, including classification methods, evaluation metrics and so on. After 2003, several systems were proposed: UbiCrawler [6] is the first crawling system announced to be deployed on WAN. The consistent hashing [7] method it adopts guarantees the load balancing among crawlers.…”

Section: Related Workmentioning

confidence: 99%

Exploring Web Partition in DHT-Based Distributed Web Crawling

Xiao

Zhang

et al. 2010

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYThe basic requirements of the distributed Web crawling systems are: short download time, low communication overhead and balanced load which largely depends on the systems' Web partition strategies. In this paper, we propose a DHT-based distributed Web crawling system and several DHT-based Web partition methods. First, a new system model based on a DHT method called the Content Addressable Network (CAN) is proposed. Second, based on this model, a network-distance-based Web partition is implemented to reduce the crawler-crawlee network distance in a fully distributed manner. Third, by utilizing the locality on the link space, we propose the concept of link-based Web partition to reduce the communication overhead of the system. This method not only reduces the number of inter-links to be exchanged among the crawlers but also reduces the cost of routing on the DHT overlay. In order to combine the benefits of the above two Web partition methods, we then propose 2 distributed multi-objective Web partition methods. Finally, all the methods we propose in this paper are compared with existing system models in the simulated experiments under different datasets and different system scales. In most cases, the new methods show their superiority.

show abstract

Section: Related Workmentioning

confidence: 99%

Exploring Web Partition in DHT-Based Distributed Web Crawling

Xiao

Zhang

et al. 2010

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…The ordinates are calculated through Eq. (6). N represents the number of Web hosts crawled by the system; D i represents the total data size downloaded from host i; T i represents the total download time of host i which is fixed to 15 minutes.…”

Section: Rtt-best (Rtt-top-1)mentioning

confidence: 99%

“…The execution time (ms) of each Web host is mapped to steps by dividing the execution time by 2 · 10 4 . For example, assuming that for one Web page the RTT = 100 ms, DLT = 100 ms, PWT = 100 ms, the total download time (execution time) of the Web host (or piece) with 10000 pages is (100+100+100)·10000 = 3·10 6 . Then the execution steps is 3·10 6 2·10 4 = 150.…”

Section: Simulation Setupsmentioning

confidence: 99%

“…[11] first proposed a series of basic concepts on paralle crawling and distributed crawling, including classification methods, evaluation metrics and so on. After 2003, several systems were proposed: UbiCrawler [6] is the first crawling system announced to be deployed on WAN. The consistent hashing [7] method it adopts guarantees the load balancing among crawlers.…”

Section: Related Workmentioning

confidence: 99%

“…Ubicrawler [6] It is the first published system to propose the idea of hash mapping the Web sites to slots (crawlers) in order to partition the set of Web hosts. The system adopts consistent hashing mechanism [7] so that the addition or removal of one slot (crawler) does not significantly change the overall mapping of Web hosts to slots (crawlers).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficient Distributed Web Crawling Utilizing Internet Resources

Xiao

Zhang

et al. 2010

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYInternet computing is proposed to exploit personal computing resources across the Internet in order to build large-scale Web applications at lower cost. In this paper, a DHT-based distributed Web crawling model based on the concept of Internet computing is proposed. Also, we propose two optimizations to reduce the download time and waiting time of the Web crawling tasks in order to increase the system's throughput and update rate. Based on our contributor-friendly download scheme, the improvement on the download time is achieved by shortening the crawlercrawlee RTTs. In order to accurately estimate the RTTs, a network coordinate system is combined with the underlying DHT. The improvement on the waiting time is achieved by redirecting the incoming crawling tasks to light-loaded crawlers in order to keep the queue on each crawler equally sized. We also propose a simple Web site partition method to split a large Web site into smaller pieces in order to reduce the task granularity. All the methods proposed are evaluated through real Internet tests and simulations showing satisfactory results.

show abstract