A Hybrid Revisit Policy For Web Search

Abstract

<p>A crawler is a program that retrieves and stores pages from the Web, commonly for a Web search engine. A crawler often has to download hundreds of millions of pages in a short period of time and has to constantly monitor and refresh the downloaded pages. Once the crawler has downloaded a significant number of pages, it has to start <em>revisiting </em>the downloaded pages in order to refresh the downloaded collection. Due to resource constraints, search engines usually have difficulties keeping the entire local repository synchronized with the web. Given the size of web today and inherent resource constraints: re-crawling too frequently leads to wasted bandwidth, re-crawling too infrequently brings down the quality of the search engine. In this paper a hybrid approach is build on the basis of which a web crawler maintains the retrieved pages &ldquo;fresh&rdquo; in the local collection. Towards this goal the concept of Page rank and Age of a web page is used. As higher page rank means that more number of users are visiting that very web page and that page has higher link popularity. Age of web page is a measure that indicates how outdated the local copy is. Using these two parameters a hybrid approach is proposed that can identify important pages at the early stage of a crawl, and the crawler re-visit these important pages with higher priority.</p

Efficient watcher based web crawler design

“…The crawlers were allowed to crawl all web sites under the domain of "emu.edu.tr" and the extracted outer links (if any). The number of web pages that have been processed by these crawlers were recorded in Table V 3.5 Experiment 5: the performance of the re-visiting policies In this experiment, the re-visiting performance has been investigated for the WBC and the following three re-visiting policies: the uniform (Pichler et al, 2011;Bhute and Meshram, 2010;Leng et al, 2011;Sharma et al, 2012;Singh and Vikasn, 2014); the proportional by rank (Pichler et al, 2011;Bhute and Meshram, 2010;Leng et al, 2011); and the proportional by top N levels (Pichler et al, 2011;Cho et al, 2012). This experiment was repeated for seven days where one crawler with five threads was used for each policy.…”

“…(1) Uniform policy: in this policy, the entire web sites are downloaded at each visit (Bhute and Meshram, 2010;Pichler et al, 2011;Leng et al, 2011;Sharma et al, 2012;Singh and Vikasn, 2014). Although this approach enriches the databases, it requires a large processing time.…”