Inverted indexes for phrases and strings

Patil, Manish; Thankachan, Sharma V.; Shah, Rahul; Hon, Wing-Kai; Vitter, Jeffrey Scott; Chandrasekaran, Sabrina

doi:10.1145/2009916.2009992

Cited by 43 publications

(29 citation statements)

References 47 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Contributions came from four continents and seven countries. At least six teams (Team 1, 3-5, 7-9) published highly influential papers on string matching problems before [15,19,22,24,26,28,33], while two teams (Team 2 and Team 6) can be considered as newcomers. As Table 1 shows, the techniques used cover a broad range and thus subsume a large fraction of previous research in k-approximate string matching.…”

Section: Competition and Methodologymentioning

confidence: 99%

State-of-the-art in string similarity search and join

et al. 2014

Self Cite

View full text Add to dashboard Cite

String similarity search and its variants are fundamental problems with many applications in areas such as data integration, data quality, computational linguistics, or bioinformatics. A plethora of methods have been developed over the last decades. Obtaining an overview of the state-of-the-art in this field is difficult, as results are published in various domains without much cross-talk, papers use different data sets and often study subtle variations of the core problems, and the sheer number of proposed methods exceeds the capacity of a single research group. In this paper, we report on the results of the probably largest benchmark ever performed in this field. To overcome the resource bottleneck, we organized the benchmark as an international competition, a workshop at EDBT/ICDT 2013. Various teams from different fields and from all over the world developed or tuned programs for two crisply defined problems. All algorithms were evaluated by an external group on two machines. Altogether, we compared 14 different programs on two string matching problems (k-approximate search and k-approximate join) using data sets of increasing sizes and with different characteristics from two different domains. We compare programs primarily by wall clock time, but also provide results on memory usage, indexing time, batch query effects and scalability in terms of CPU cores. Results were averaged over several runs and confirmed on a second, different hardware platform. A particularly interesting observation is that disciplines can and should learn more from each other, with the three best teams rooting in computational linguistics, databases, and bioinformatics, respectively.

show abstract

Section: Competition and Methodologymentioning

confidence: 99%

State-of-the-art in string similarity search and join

et al. 2014

Self Cite

View full text Add to dashboard Cite

show abstract

“…Their index, on the other hand, turns out to be very competitive in practice. There exist several other indexes of practical interest [22,6,23,24,25].…”

Section: Sourcementioning

confidence: 99%

New space/time tradeoffs for top- k document retrieval on sequences

Navarro

Thankachan

2014

Theoretical Computer Science

Self Cite

View full text Add to dashboard Cite

We address the problem of indexing a collection D = {T 1 , T 2 , ...T D } of D string documents of total length n, so that we can efficiently answer top-k queries: retrieve k documents most relevant to a pattern P of length p given at query time.There exist linear-space data structures, that is, using O(n) words, that answer such queries in optimal O(p + k) time for an ample set of notions of relevance. However, using linear space is not sufficiently good for large text collections. In this paper we explore how far the space/time tradeoff for this problem can be pushed. We obtain three results: (1) When relevance is measured as term frequency (number of times P appears in a document T i ), an index occupying |CSA|+o(n) bits answers the query in time O(t search (p)+k lg 2 k lg ε n), where CSA is a compressed suffix array indexing D, t search is its time to find the suffix array interval of P, and ε > 0 is any constant. (2) With the same measure of relevance, an index occupying |CSA| + n lg D + o(n lg σ + n lg D) bits answers the query in time O(t search (p) + k lg * k), where lg * k is the iterated logarithm of k. (3) When the relevance depends only on the documents, an index occupying |CSA| + O(n lg lg n) bits answers the query in O(t search (p) + k t SA ) time, where t SA is the time the CSA needs to retrieve a suffix array cell. On our way, we obtain some other results of independent interest.

show abstract

“…We compare their best performing variant, GREEDY, in this paper. Culpepper et al [5] adapted the scheme to large natural language text collections (where each word is taken as an atomic symbol), showing that it was competitive with inverted indexes for some queries (see previous work on this line by Patil et al [20]). The seminal work of Hon et al [13] also included succinct variants, which were implemented by Navarro and Valenzuela [19] on top of a compressed representation of D.…”

Section: Basic Conceptsmentioning

confidence: 99%

Improved Single-Term Top-k Document Retrieval

Gog

Navarro

2014

2015 Proceedings of the Seventeenth Workshop on Algorithm Engineering and Experiments (ALENEX)

View full text Add to dashboard Cite

On natural language text collections, finding the k documents most relevant to a query is generally solved with inverted indexes. On general string collections, however, more sophisticated data structures are necessary. Navarro and Nekrich [SODA 2012] showed that a linear-space index can solve such top-k queries in optimal time O(m + k), where m is the query length. Konow and Navarro [DCC 2013] implemented the scheme, managing to solve top-k queries within microseconds with an index using 3.3-4.0 bytes per character (this includes the storage of the collection itself). In this paper we introduce a new implementation using significantly less space, 2.5-3.0 bytes per character (again, including the collection), and retaining similar query times. For short queries, which are the most difficult, our new index actually outperforms the previous one, as well as all the other solutions in the literature. We also show that our index can be built on very large text collections, and that it can handle phrase queries efficiently on natural language text collections. In the latter case, it uses about the same space of the tokenized text (and replaces it), while answering phrase queries an order of magnitude faster than a positional inverted index.

show abstract

Inverted indexes for phrases and strings

Cited by 43 publications

References 47 publications

State-of-the-art in string similarity search and join

State-of-the-art in string similarity search and join

New space/time tradeoffs for top- k document retrieval on sequences

Improved Single-Term Top-k Document Retrieval

Contact Info

Product

Resources

About