Inverted files for text search engines

Zobel, Justin; Moffat, Alistair

doi:10.1145/1132956.1132959

Cited by 906 publications

(702 citation statements)

References 140 publications

Supporting

Mentioning

681

Contrasting

Unclassified

Order By: Relevance

“…The most popular are inverted indexes [5,10,68]. Inverted indexes are efficient because their search strategy is based on the vocabulary (the set of distinct words in the text), which is usually much smaller than the text, and thus fits in main memory.…”

Section: Indexed Text Searchingmentioning

confidence: 99%

Distributed text search using suffix arrays

Arroyuelo¹,

Bonacic²,

Gil-Costa³

et al. 2014

Parallel Computing

View full text Add to dashboard Cite

Text search is a classical problem in Computer Science, with many data-intensive applications. For this problem, suffix arrays are among the most widely known and used data structures, enabling fast searches for phrases, terms, substrings and regular expressions in large texts. Potential application domains for these operations include large-scale search services, such as Web search engines, where it is necessary to efficiently process intensive-traffic streams of on-line queries. This paper proposes strategies to enable such services by means of suffix arrays. We introduce techniques for deploying suffix arrays on clusters of distributed-memory processors and then study the processing of multiple queries on the distributed data structure. Even though the cost of individual search operations in sequential (non-distributed) suffix arrays is low in practice, the problem of processing multiple queries on distributed-memory systems, so that hardware resources are used efficiently, is relevant to services aimed at achieving high query throughput at low operational costs. Our theoretical and experimental performance studies show that our proposals are suitable solutions for building efficient and scalable on-line search services based on suffix arrays. IntroductionIn the last decade, the design of efficient data structures and algorithms for textual databases and related applications has received a great deal of attention, due to the rapid growth of the amount of text data available from different sources. Typical applications support text searches over big text collections in a client-server fashion, where the user queries are answered by a dedicated server [15]. The server efficiency-in terms of running time-is of paramount importance in cases where the services demanded by clients generate a heavy work load. A feasible way to overcome the limitations of sequential computers is to resort to the use of several computers, or processors, which work together to serve the ever increasing client demands [19].One such approach to efficient parallelization is to distribute the data onto the processors, in such a way that it becomes feasible to exploit locality via parallel processing of user requests, each on a subset of the data. As opposed to shared-memory models, this distributed-memory model provides the benefit of better * Corresponding author. Address: Av. España 1680, Valparaíso, Chile. Phone: +56 2 432 6722. Fax: +56 2 432 6702. , in distributed memory systems, and describes strategies to reduce the inter-processor communication and to improve the load balance at search time. Indexed Text SearchingThe advent of powerful processors and cheap storage has enabled alternative models for information retrieval, other than the traditional one of a collection of documents indexed by a fixed set of keywords. One is the full text model, in which the user expresses its information need via words, phrases or patterns to be matched for, and the information system retrieves those documents containing the user-specified pa...

show abstract

Section: Indexed Text Searchingmentioning

confidence: 99%

Distributed text search using suffix arrays

Arroyuelo¹,

Bonacic²,

Gil-Costa³

et al. 2014

Parallel Computing

View full text Add to dashboard Cite

show abstract

“…In the STR-tree, a leaf node includes entries in the form (optr, loc, oti), where optr is a pointer to an object in D, oti represents the tag information of an object, which is indexed by inverted lists [12]. A intermediate node contains these entries in the form (N ptr, MBR, Ntsum), where Ntsum represents the tag summary information of its child nodes referred by Nptrs.…”

Section: Str-tree: a Refined Hybrid Indexing Mechanismmentioning

confidence: 99%

“…Finally, all of node sets in I k are appended to list L(line 11). After all node sets L are found, we check each node set in L to see whether its MinRank score is less than uppC (line [12][13][14]. Those that cannot qualify the conditions are eliminated.…”

Section: Fig 3 the Construction Of Shadow Prefix-treementioning

confidence: 99%

Co-spatial Searcher: Efficient Tag-Based Collaborative Spatial Search on Geo-social Network

Zhang

Meng

Zhou

et al. 2012

Database Systems for Advanced Applications

View full text Add to dashboard Cite

Abstract. The proliferation of geo-social network, such as Foursquare and Facebook Places, enables users to generate location information and its corresponding descriptive tags. Using geo-social networks, users with similar interests can plan for social activities collaboratively. This paper proposes a novel type of query, called Tag-based top-k Collaborative Spatial (TkCoS) query, for users to make outdoor plans collaboratively. This type of queries aim to retrieve groups of geographic objects that can satisfy a group of users' requirements expressed in tags, while ensuring that the objects be within the minimum spatial distance from the users. To answer TkCoS queries efficiently, we introduce a hybrid index structure called Spatial-Tag R-tree (STR-tree), which is an extension of the R-tree. Based on STR-tree, we propose a query processing algorithm that utilizes both spatial and tag similarity constraints to prune search space and identify desired objects quickly. Moreover, a differential impact factor is adopted to fine-tune the returned results in order to maximize the users' overall satisfaction. Extensive experiments on synthetic and real datatsets validate the efficiency and the scalability of the proposed algorithm.

show abstract

“…As mentioned above, the multimap abstract data type is related to the inverted file and inverted index structures, which are well-known in text indexing applications (e.g., see Knuth [20]) and are also used in search engines (e.g., see Zobel and Moffat [31]). Cutting and Pedersen [12] describe an inverted file implementation that uses B-trees for the indexing structure and supports insertions, but doesn't support deletions efficiently.…”

Section: Introductionmentioning

confidence: 99%

Cache-Oblivious Dictionaries and Multimaps with Negligible Failure Probability

Goodrich

Hirschberg

Mitzenmacher

et al. 2012

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. A dictionary (or map) is a key-value store that requires all keys be unique, and a multimap is a key-value store that allows for multiple values to be associated with the same key. We design hashing-based indexing schemes for dictionaries and multimaps that achieve worst-case optimal performance for lookups and updates, with minimal space overhead and sub-polynomial probability that the data structure will require a rehash operation. Our dictionary structure is designed for the Random Access Machine (RAM) model, while our multimap implementation is designed for the cache-oblivious external memory (I/O) model. The failure probabilities for our structures are sub-polynomial, which can be useful in cryptographic or data-intensive applications.

show abstract

Inverted files for text search engines

Cited by 906 publications

References 140 publications

Distributed text search using suffix arrays

Distributed text search using suffix arrays

Co-spatial Searcher: Efficient Tag-Based Collaborative Spatial Search on Geo-social Network

Cache-Oblivious Dictionaries and Multimaps with Negligible Failure Probability

Contact Info

Product

Resources

About