Adaptive on-line page importance computation

Abstract

The computation of page importance in a huge dynamic graph has recently attracted a lot of attention because of the web. Page importance, or page rank is defined as the fixpoint of a matrix equation. Previous algorithms compute it off-line and require the use of a lot of extra CPU as well as disk resources (e.g. to store, maintain and read the link matrix). We introduce a new algorithm OPIC that works on-line, and uses much less resources. In particular, it does not require storing the link matrix. It is on-line in that it continuously refines its estimate of page importance while the web/graph is visited. Thus it can be used to focus crawling to the most interesting pages. We prove the correctness of OPIC. We present Adaptive OPIC that also works on-line but adapts dynamically to changes of the web. A variant of this algorithm is now used by Xyleme.We report on experiments with synthetic data. In particular, we study the convergence and adaptiveness of the algorithms for various scheduling strategies for the pages to visit. We also report on experiments based on crawls of significant portions of the web.

Query-Log Based Authority Analysis for Web Information Search

“…Thereby precision is computed for each point in the ranking where a relevant document occurs. For the above left ranking this is 1 2 , 1 2 , 1 2 , and for the right one we get 1 4 , 1 4 , 3 10 , which averages to 0.5, repectively 0.26. When comparing different retrieval algorithms one often wants to average the precisions for several queries.…”

“…An excerpt of a browser history is shown in Table 1. 1. Here the user query "france highest mountain" is refined by the query "france highest summit" and a page about the Mont Blanc is clicked.…”

“…A Markov chain with discrete parameter space is 1. homogeneous, if the transition probabilities p ij = P [X n+1 = j|X n = i] are independent of n.…”

“…Beyond the two most prominent approaches to link analysis, the HITS method by Kleinberg [30] and PageRank by Brin and Page [11,12] (for a description see Section 2.2.3, respectively 2.2.2), a variety of extensions and generalizations of both methods has been developed [6,9,31,1,18,23,14]. Some of these shall be discussed in Section 3.1.…”

On Approximation Algorithms for Data Mining Applications

“…Other work on measuring the importance of web pages include [1]. Other research directions for mining useful information from the web include [23], where the web is searched for frequent itemsets by a method using features of the algorithm for dynamic itemset counting [21].…”