Query forwarding in geographically distributed search engines

Abstract

Query forwarding is an important technique for preserving the result quality in distributed search engines where the index is geographically partitioned over multiple search sites. The key component in query forwarding is the thresholding algorithm by which the forwarding decisions are given. In this paper, we propose a linear-programming-based thresholding algorithm that significantly outperforms the current state-of-the-art in terms of achieved search efficiency values. Moreover, we evaluate a greedy heuristic for partial index replication and investigate the impact of result cache freshness on query forwarding performance. Finally, we present some optimizations that improve the performance further, under certain conditions. We evaluate the proposed techniques by simulations over a real-life setting, using a large query log and a document collection obtained from Yahoo!. Categories and Subject Descriptors General TermsAlgorithms, Design, Performance, Experimentation KeywordsSearch engines, distributed IR, query forwarding, optimization, linear programming, index replication, result caching BACKGROUNDCommercial web search engines of the past relied on a single search site (data center), which processed queries issued Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. from all around the world. This approach had the typical scalability problems in centralized architectures. Moreover, queries issued from distant locations suffered from poor response times as the network latency between the user and the site became an issue. For such queries, either the query processing times had to be shortened, thus degrading the result quality, or users experienced unreasonable response times, which had implications on user satisfaction [16].At this point, replicating the data (i.e., the web collection and the inverted index built upon it) over multiple, geographically distant search sites emerged as a feasible solution. In this strategy, each geographical region is mapped to a nearby search site. A search site processes over its full web index only the queries originating from the regions assigned to itself 1 . Although this strategy reduces network latencies, the scalability still remains as an issue since the entire web index had to be maintained on all search sites and queries are evaluated over the full web index.A strategy that contrasts replication is to partition the data disjointly and assign each site only the documents obtained (crawled) from its region [8]. In this strategy, local queries of a region are evaluated over the partial index in the corresponding search site. The underlying assumption here is that users are interested more in documents located...

Energy-price-driven query processing in multi-center web search engines

“…We simulate a web search engine with five data centers, denoted as DC-A, DC-B, DC-C, DC-D, and DC-E. Data centers are assumed to be located in capital cities of five geographically distant countries, which are not disclosed due to the risk of revealing financially sensitive information about query traffic volumes. Network latencies between data centers as well as those between users and data centers are estimated by applying the technique described in [8]. We assume that the building blocks of data centers are identical, homogeneous search clusters.…”

“…A few works investigate the performance of multi-center web search engines [1,7,8]. Cambazoglu et al [7] try to quantify performance benefits of such search engines.…”

“…Their work provides an algorithm for forwarding queries between data centers to maintain the quality of search results, relative to that of a centralized system. Cambazoglu et al [8] propose an improved query forwarding algorithm, using linear programming. Both works have indirect consequences on reducing the energy cost, as they reduce the number of data centers involved in query processing.…”

A Focused Crawler URL Analysis Algorithm based on Semantic Content and Link Clustering in Cloud Environment

“…Vertical search engines provide Web query based on semantic content with domain knowledge in a given industry or theme [1]. Focused crawlers make more satisfactory results than general search engines on a small scale of data to meet the needs of specific users, because general search engines have many limitation such as low coverage ratio [2].…”

Assigning documents to master sites in distributed search

“…Upon receiving a user query, a site either processes a query locally or forwards it to other sites for processing. Techniques for deciding when to forward a query are consequently critical to the performance of such a distributed search engine [1,7].…”