Document allocation policies for selective searching of distributed indexes

Abstract

Indexes for large collections are often divided into shards that are distributed across multiple computers and searched in parallel to provide rapid interactive search. Typically, all index shards are searched for each query. For organizations with modest computational resources the high query processing cost incurred in this exhaustive search setup can be a deterrent to working with large collections. This paper investigates document allocation policies that permit searching only a few shards for each query (selective search) without sacrificing search accuracy. Random, source-based and topic-based document-to-shard allocation policies are studied in the context of selective search.A thorough study of the tradeoff between search cost and search accuracy in a sharded index environment is performed using three large TREC collections. The experimental results demonstrate that selective search using topic-based shards cuts the search cost to less than 1/5th of that of the exhaustive search without reducing search accuracy across all the three datasets. Stability analysis shows that 90% of the queries do as well or improve with selective search. An overlap-based evaluation with an additional 1000 queries for each dataset tests and confirms the conclusions drawn using the smaller TREC query sets.

A Distributed Framework for Content Search Using Small World Communities

“…However, in this approach one step goes forward, using an ontology-based fuzzy similarity, based on both semantic and structural issues. The centralization used in these techniques [28][29][30][31] for dealing with a distributed framework results in a reduction of the search efficiency. Obviously, gathering information within the distributed system causes a computational overload.…”

Evaluating non-deterministic retrieval systems

“…Recent research has focused on reducing the search cost per query without hurting overall effectiveness by reordering the documents in each shard by topic or similarity [3]. These systems are able to achieve effectiveness close to a search over the entire collection (exhaustive search) while using only a few shards for each Table 1: The proportion of system instances that demonstrated a significant difference using a paired t-test, and the p values when comparing the sample-based IR algorithm proposed by Kulkarni and Callan [3] at varying CSI sample rates with a deterministic exhaustive search, and with itself (a nondeterministic algorithm) with a CSI sample rate of 4% using the TREC GOV2 dataset and TREC topics 701 -850.…”

“…Thus, we have 10 different instances of the sharded index. As with the original experiments [3], 50 shards were formed per instance, and the full dependency model (FDM) is used to rank the queries [4]. Selecting a subset of 5 shards produced equivalent retrieval results at depth 10 to exhaustive search [3].…”

“…with non-deterministic output introduces an additional dimension of variability into this scenario. In this paper, we contribute the following • We present a methodology to solve the two-dimensional significance testing problem (Section 3); • We explore the properties of our solution on a case study of common sampling-based algorithms -shard construction and centralized resource allocation in distributed IR [3,6]. We examine the variability that can occur in this environment, observing that an apparently significant result on one instance of a samplebased algorithm can be contradicted by another, and we demonstrate the use of our two-dimensional significance testing methods to handle the variability and provide sound statistical inferences (Section 4).…”

Balanced Search Space Partitioning for Distributed Media Redundant Indexing

“…e goals of index partitioning algorithms are to distribute documents across nodes based on document similarity, to facilitate the e cient selection of retrieval resources, such that documents relevant to a query are concentrated across a few shards [22]. ere are two main index partitioning strategies [9]:…”