Document Filtering for Fast Ranking

Persin, Michael

doi:10.1007/978-1-4471-2099-5_35

Cited by 46 publications

(49 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the calculation of the similarity values instead of an accumulator array dynamic data structures can be used for memory efficiency [Witten, 1994]. A partial query evaluation or pruning strategy and its effectiveness and efficiency should also be investigated [Brown, 1995;Moffat, Zobel, 1996;Persin 1994]. Our scalability experiments (Table XIII) and the results reported in [Moffat, Zobel, 1996] imply that when our ICsIIS approach is combined with the restricted accumulators (quit and continue methods of Moffat and Zobel), it can further improve the efficiency performance.…”

Section: Discussionmentioning

confidence: 95%

“…As a result, the efficiency of FS decreases with increasing query length, since for each query term another posting list must be processed. It is possible to employ a partial evaluation (or pruning) strategy that skips some of the query terms to improve search efficiency with similar search effectiveness [Buckley, Lewit, 1985;Brown, 1995;Persin, 1994;Moffat, Zobel, 1996].…”

Section: Previous and Related Workmentioning

confidence: 99%

“…There are various optimization techniques used for inverted index searches [Buckley, Lewit, 1985;Persin, 1994;Brown 1995;Moffat, Zobel, 1996]. They aim to use only the most informative parts of inverted list and try to increase efficiency of query processing without F. Can, İ. S. Altıngövde, E. Demir: Efficiency and effectiveness of query processing p. 3 deteriorating retrieval effectiveness.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Efficiency and effectiveness of query processing in cluster-based retrieval

Can

Altıngövde

Demir

2004

Information Systems

View full text Add to dashboard Cite

Our research shows that for large databases, without considerable additional storage overhead, clusterbased retrieval (CBR) can compete with the time efficiency and effectiveness of the inverted indexbased full search (FS). The proposed CBR method employs a storage structure that blends the cluster membership information into the inverted file posting lists. This approach significantly reduces the cost of similarity calculations for document ranking during query processing and improves efficiency. For example, in terms of in-memory computations, our new approach can reduce query processing time to 39% of FS. The experiments confirm that the approach is scalable and system performance improves with increasing database size. In the experiments, we use the Cover Coefficient-based Clustering Methodology (C 3 M), and the Financial Times database of TREC-4 containing 210,158 documents of size 564 MB defined by 229,748 terms with total of 29,545,234 inverted index elements. This study provides CBR efficiency and effectiveness experiments using the largest corpus in an environment that employs no user interaction or user behavior assumption for clustering.

show abstract

Section: Discussionmentioning

confidence: 95%

Section: Previous and Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Efficiency and effectiveness of query processing in cluster-based retrieval

Can

Altıngövde

Demir

2004

Information Systems

View full text Add to dashboard Cite

show abstract

“…Despite various attempts to displace inverted indexes from their dominant position for document ranking tasks over the years, no alternative has been able to consistently produce the same level of efficiency, effectiveness, and time / space trade-offs that inverted indexes can provide (see, for instance Zobel et al [45]). Ranked document retrieval requires that only the top-k documents are returned, and, as a result, researchers have proposed many heuristic approaches to improve top-k efficiency [1,4,5,6,32,38]. These approaches can be classified in two general categories: term-at-a-time (TAAT) and document-at-a-time (DAAT).…”

Section: Inverted Indexesmentioning

confidence: 99%

Efficient in-memory top-k document retrieval

Culpepper

Petri

Scholer

2012

Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

For over forty years the dominant data structure for ranked document retrieval has been the inverted index. Inverted indexes are effective for a variety of document retrieval tasks, and particularly efficient for large data collection scenarios that require disk access and storage. However, many efficiency-bound search tasks can now easily be supported entirely in-memory as a result of recent hardware advances.In this paper we present a hybrid algorithmic framework for inmemory bag-of-words ranked document retrieval using a self-index derived from the FM-Index, wavelet tree, and the compressed suffix tree data structures, and evaluate the various algorithmic trade-offs for performing efficient queries entirely in-memory. We compare our approach with two classic approaches to bag-of-words queries using inverted indexes, term-at-a-time (TAAT) and document-at-atime (DAAT) query processing. We show that our framework is competitive with state-of-the-art indexing structures, and describe new capabilities provided by our algorithms that can be leveraged by future systems to improve effectiveness and efficiency for a variety of fundamental search operations.

show abstract

“…The use of filters [22] is an evaluation technique that allows to recognize what documents are probable of being ranked, reducing the main memory volume required and the queries evaluation time, without harming the effectiveness of results. Because of that, we use the filtering technique proposed by Persin [21] to discard the irrelevant documents for the queries. It uses two filters: the f ins to reduce the amount of memory used and f add to reduce the CPU time.…”

Section: Distributed Inverted Files and Vector Space Modelmentioning

confidence: 99%