Effective Change Detection Using Sampling

Abstract

For a large-scale data-intensive environment, such as the World-Wide Web or data warehousing, we often make local copies of remote data sources. Due to limited network and computational resources, however, it is often difficult to monitor the sources constantly to check for changes and to download changed data items to the copies. In this scenario, our goal is to detect as many changes as we can using the fixed download resources that we have. In this paper we propose three sampling-based download policies that can identify more changed data items effectively. In our sampling-based approach, we first sample a small number of data items from each data source and download more data items from the sources with more changed samples. We analyze the effectiveness of the sampling-based policies and compare our proposed policies to existing ones, including the state-of-the-art frequency-based policy in [7,10]. Our experiments on synthetic and real-world data will show the relative merits of various policies and the great potential of our sampling-based policy. In certain cases, our sampling-based policy could download twice as many changed items as the best existing policy.

Mining for Information Discovery on the Web: Overview and Illustrative Research

“…Several works have developed techniques to crawl the Web efficiently, and to detect and model changes at pages so that the crawler can focus on pages that change frequently [20]. For domain-specific search engines, focused crawling to retrieve Web pages in the domain has been studied [12,71,91], as we mentioned in Section 2.1.…”

Temporal Ranking of Search Engine Results

“…The concept of continual queries was conceived to alleviate the impact of changing web content on information retrieval (e.g., [15,7]). Continual query systems guarantee the freshness of web pages by continuously monitoring them with appropriate frequencies.…”

Scalable application-aware data freshening

“…The problem of managing the freshness of local copies has been explored in the web caching area [19]. Here, however, the model of synchronization is 6 Euclidean Distance between two elements is…”

“…We assume that it is possible to obtain the number of updates to an element over some time period. Prior work has shown how the source can use estimation [4] and sampling [6] techniques to obtain a good estimate of these update frequencies. These frequency estimates would be periodically communicated to the mirror.…”

“…For pull-based approaches, clients decide on a refresh schedule based on knowledge of the change frequency of documents. This information can be obtained using Time-To-Live(TTL) [7] information, application of a probabilistic distributions [4], or sampling from servers [6]. A TTL represents the estimated period a web document will remain fresh and is widely used for web cache consistency.…”