Focused Crawls, Tunneling, and Digital Libraries

Abstract

Crawling the Web to build collections of documents related to pre-specified topics became an active area of research during the late 1990's, crawler technology having been developed for use by search engines. Now, Web crawling is being seriously considered as an important strategy for building large scale digital libraries. This paper covers some of the crawl technologies that might be exploited for collection building. For example, to make such collection-building crawls more effective, focused crawling was developed, in which the goal was to make a "best-first" crawl of the Web. We are using powerful crawler software to implement a focused crawl but use tunneling to overcome some of the limitations of a pure best-first approach. Tunneling has been described by others as not only prioritizing links from pages according to the page's relevance score, but also estimating the value of each link and prioritizing them as well. We add to this mix by devising a tunneling focused crawling strategy which evaluates the current crawl direction on the fly to determine when to terminate a tunneling activity. Results indicate that a combination of focused crawling and tunneling could be an effective tool for building digital libraries.

A Frame Work for Topical Collections Make with Focused and Accelerated Focused Crawlers

“…Web surfing is feasible due to the fact that most pages link to similar pages. Some recent work by Menczer [3,4] provides interesting insights into the relationship between content similarities and relatedness among Web pages. He finds that both content and links provide a weak yet significant signal about the (semantic) relatedness of Web pages [18][19].…”

“…On other occasions we find very valuable and accurate information. Hence, the large size, dynamism, and uncontrolled nature of the Web offer new challenges for information handling, retrieval, and usage [4][5][6].…”

CT-FC: more Comprehensive Traversal Focused Crawler

“…Quadrant II (connected indirectly and in a forward direction search) contains relevant documents, which have indirectly connected characteristic, i.e. connected through one or several irrelevant documents [9], [10], [11]. Relevant documents in quadrant III connected directly through in-links of downloaded documents.…”

Enhanced Search Scheme Precision and Performance using a GA Approach with Application to Arabic Content

“…In this algorithm, the heuristics (based on previous search results) are employed in the search ranking and queue order. Non-promising Universal Resource Locators (URLs) are placed in the back of the queue, where they rarely get a chance to be visited (Bergmark, 2002;Bergmark et al, 2002;Chakrabarti et al, 2007). Obviously, this type of search algorithm is more common than the breadth-first search algorithm since it examines the relevant page locations and avoids retrieving non-related pages.…”