A cross-language focused crawling algorithm based on multiple relevance prediction strategies

Abstract

a b s t r a c tFocused crawling is increasingly seen as a solution to address the scalability limitations of existing general-purpose search engines, by traversing the Web to only gather pages that are relevant to a specific topic. How to predict the relevance of the unvisited pages pointed to by candidate URLs in the crawling frontier to a given topic is a key issue in the design of focused crawlers. In this paper, we propose a novel approach based on multiple relevance prediction strategies to address this problem. For cross-language crawling, we first introduce a hierarchical taxonomy to describe topics in both English and Chinese. We then present a formal description of the relevance predicting process and discuss four strategies that make use of page contents, anchor texts, URL addresses and link types of Web pages, respectively, to evaluate the relevance more accurately, in which we propose a particular strategy using Chinese URL addresses to estimate the relevance of cross-language Web pages. Finally, we get a new focused crawling algorithm (FCMRPS, Focused Crawling based on Multiple Relevance Prediction Strategies) based on the combination of these strategies and Shark-Search, which is a classic focused crawling algorithm. Experiments show that the FCMRPS is more effective than the traditional algorithms, namely BreadthFirst, Best-First and Shark-Search, in terms of precision and sum of information.

An Enhanced Semantic Focused Web Crawler Based on Hybrid String Matching Algorithm

“…Most of the focused web crawlers, [2][3][4] have used only full page text to estimate the similarity score of the web page, [5][6][7][8] used both full page text and the anchor text for estimating the relevance score. The authors [9][10][11] used cosine similarity metric to estimate the similarity score of the unvisited web pages. The cosine similarity value can be assessed by finding the Term Frequency and Inverse Document Frequency (TF-IDF).…”

A Critique Empirical Evaluation of Relevance Computation for Focused Web Crawlers

“…Only full page text and anchor text cannot capture the similarity of the web page accurately. To overcome this issue [9] proposed a focused crawler for cross language crawling, which adopts an algorithm, known as Focused crawling for Multiple Relevance Prediction Strategies (FCMRPS). The FCMRPS is an integration of the average similarity score of four target variables (full page text, anchor text, URL address and link structure) with the topic and shark search algorithm.…”

A new fuzzy-based method to weigh the related concepts in semantic focused web crawlers

“…Focused crawling based on strategies of preceding multiple relevance [12] uses several strategies of predicting relevance to compute the relation of unvisited pages with user's topic. The ideas of focused crawler algorithm presented consist of combination of four strategies which are applied with the hierarchical taxonomy of topics to predict the relevance more accurately.…”