Estimating frequency of change

Abstract

Many online data sources are updated autonomously and independently. In this article, we make the case for estimating the change frequency of data to improve Web crawlers, Web caches and to help data mining. We first identify various scenarios, where different applications have different requirements on the accuracy of the estimated frequency. Then we develop several "frequency estimators" for the identified scenarios, showing analytically and experimentally how precise they are. In many cases, our proposed estimators predict change frequencies much more accurately and improve the effectiveness of applications. For example, a Web crawler could achieve 35% improvement in "freshness" simply by adopting our proposed estimator.

Improving the Quality of Web Archives through the Importance of Changes

“…Pandey and Olston [13] propose a recrawl scheduling strategy based on information longevity to improve the freshness of web pages. In [8], Cho et al estimate the frequency of page changes based on the Poisson process. In other studies [6,7], they propose efficient policies to improve the freshness of web pages.…”

“…Frequency [7] selects pages to be archived according to their frequency of changes estimated by the Poisson model [8]. Hot pages that change too often are penalized to maximize the freshness of pages.…”

“…They consider the crawl useful even if the captured version is almost equal to the previous one. Moreover, they estimate the frequency of page changes based on the homogeneous Poisson model [8,6] with a constant change rate λ. This model is valid when the time granularity of changes is longer than one month as shown in [15] which is far from being the common case.…”

Scalable application-aware data freshening

“…We assume that it is possible to obtain the number of updates to an element over some time period. Prior work has shown how the source can use estimation [4] and sampling [6] techniques to obtain a good estimate of these update frequencies. These frequency estimates would be periodically communicated to the mirror.…”

“…For pull-based approaches, clients decide on a refresh schedule based on knowledge of the change frequency of documents. This information can be obtained using Time-To-Live(TTL) [7] information, application of a probabilistic distributions [4], or sampling from servers [6]. A TTL represents the estimated period a web document will remain fresh and is widely used for web cache consistency.…”