The anatomy of web crawlers

Abstract

World Wide Web (www) is the gigantic and richest source of information. To retrieve the information from this imperative resource, Search Engines are generally used. For this purpose these Search engines rely on massive collections of web pages that have been downloaded by web crawlers. A Web crawler is a program that traverses the web by following the ever changing, dense and distributed hyperlinked structure and thereafter storing downloaded pages in a large repository which is later indexed for efficient execution of user queries. Thus, web crawlers are becoming increasingly important. Various web crawling architectures have been proposed in recent years. In this paper a survey of different architectures of web crawlers along with their comparisons has been carried out that takes into account various important features like scalability, manageability, page refresh policy, politeness policy etc.

Seed URL Selection and Performance Analysis in Web Crawlers: A Comprehensive Review

“…Artımlı tarayıcı Web'i sürekli tarar ve sayfaları periyodik olarak yeniden ziyaret eder [55]. Artımlı web tarayıcısının temel hedefi yerel koleksiyonu taze tutmak ve yerel koleksiyonun kalitesini artırmaktır [56]. Odaklanmış taramada artımlı web tarayıcılardan toplanan bilgilerin alaka düzeyi diğer tarayıcılara kıyasla az olabilmektedir.…”