Optimized Query Execution in Large Search Engines with Global Page Ordering

Long, Xiaohui; Suel, Torsten

doi:10.1016/b978-012722442-8/50020-3

Cited by 77 publications

(89 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using one setup, we measured the disk access costs of the various policies in terms of the total number of 4 KB disk block accesses. We have found that this provides a reasonable estimate of the cost of our actual query processor [24] when using one or two disks and a fast CPU. We used another setup where we preloaded inverted lists into main memory to measure the CPU costs of the methods, which is important on systems with more I/O bandwidth.…”

Section: Experimental Evaluationmentioning

confidence: 88%

“…We believe that integrating projection caching into pruning techniques such as [24] should not be difficult for two reasons: First, as discussed a projection can be treated just as any other inverted list in the index. Second, we observed that under a good choice of policies and parameters, the overhead of generating projections is tiny, and would still be small even when aggressive pruning brings down the baseline cost.…”

Section: Discussionmentioning

confidence: 99%

“…The first two factors can be easily included while computing scores as outlined above. The most commonly used way to integrate the other factors is to precompute a global importance score for each document, as done in PageRank [9], or a few importance scores for different topic groups [18], and to simply add these scores to the term-based scores during query execution [29,24,30]. Our approach does not depend on the ranking function as long as the total cost is dominated by the inverted list traversal.…”

Section: Term-based Rankingmentioning

confidence: 99%

“…There has been a significant amount of work in the IR and database communities on this issue under various scenarios; see [1,2,12,14,16,24,28,35] for recent work. Various schemes are apparently in use in current engines but details are closely guarded.…”

Section: Term-based Rankingmentioning

confidence: 99%

See 3 more Smart Citations

Three-Level Caching for Efficient Query Processing in Large Web Search Engines

Long¹,

Suel²

2006

World Wide Web

Self Cite

View full text Add to dashboard Cite

Large web search engines have to answer thousands of queries per second with interactive response times. Due to the sizes of the data sets involved, often in the range of multiple terabytes, a single query may require the processing of hundreds of megabytes or more of index data. To keep up with this immense workload, large search engines employ clusters of hundreds or thousands of machines, and a number of techniques such as caching, index compression, and index and query pruning are used to improve scalability. In particular, two-level caching techniques cache results of repeated identical queries at the frontend, while index data for frequently used query terms are cached in each node at a lower level.We propose and evaluate a three-level caching scheme that adds an intermediate level of caching for additional performance gains. This intermediate level attempts to exploit frequently occurring pairs of terms by caching intersections or projections of the corresponding inverted lists. We propose and study several offline and online algorithms for the resulting weighted caching problem, which turns out to be surprisingly rich in structure. Our experimental evaluation based on a large web crawl and real search engine query log shows significant performance gains for the best schemes, both in isolation and in combination with the other caching levels. We also observe that a careful selection of cache admission and eviction policies is crucial for best overall performance.

show abstract

Section: Experimental Evaluationmentioning

confidence: 88%

Section: Discussionmentioning

confidence: 99%

Section: Term-based Rankingmentioning

confidence: 99%

Section: Term-based Rankingmentioning

confidence: 99%

See 2 more Smart Citations

Three-Level Caching for Efficient Query Processing in Large Web Search Engines

Long¹,

Suel²

2006

World Wide Web

Self Cite

View full text Add to dashboard Cite

show abstract

“…The pruning techniques considered here (see also [4,10,21]) are heuristic in nature in that they trade off some loss in result quality (effectiveness in IR jargon) for speed without being able to predict and control the resulting effects (other than by experimentation). For the special but important case where the global score is a weighted sum of tf*idf-based text relevance and link-based, and thus query-independent, authority, additional pruning heuristics have been developed in [7,28].…”

Section: Permission To Copy Without Fee All or Part Of This Materials mentioning

confidence: 99%

Top-k Query Evaluation with Probabilistic Guarantees

Theobald

Weikum

Schenkel

et al. 2004

Proceedings 2004 VLDB Conference

View full text Add to dashboard Cite

Top-k queries based on ranking elements of multidimensional datasets are a fundamental building block for many kinds of information discovery. The best known general-purpose algorithm for evaluating top-k queries is Fagin's threshold algorithm (TA). Since the user's goal behind top-k queries is to identify one or a few relevant and novel data items, it is intriguing to use approximate variants of TA to reduce run-time costs. This paper introduces a family of approximate top-k algorithms based on probabilistic arguments. When scanning index lists of the underlying multidimensional data space in descending order of local scores, various forms of convolution and derived bounds are employed to predict when it is safe, with high probability, to drop candidate items and to prune the index scans. The precision and the efficiency of the developed methods are experimentally evaluated based on a large Web corpus and a structured data collection.

show abstract

Information retrieval on Turkish texts

Can¹,

Kocberber²,

Balcik³

et al. 2007

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

In this study, we investigate information retrieval (IR) on Turkish texts using a large-scale test collection that contains 408,305 documents and 72 ad hoc queries. We examine the effects of several stemming options and query-document matching functions on retrieval performance. We show that a simple word truncation approach, a word truncation approach that uses language-dependent corpus statistics, and an elaborate lemmatizer-based stemmer provide similar retrieval effectiveness in Turkish IR. We investigate the effects of a range of search conditions on the retrieval performance; these include scalability issues, query and document length effects, and the use of stopword list in indexing.

show abstract

Optimized Query Execution in Large Search Engines with Global Page Ordering

Cited by 77 publications

References 8 publications

Three-Level Caching for Efficient Query Processing in Large Web Search Engines

Three-Level Caching for Efficient Query Processing in Large Web Search Engines

Top-k Query Evaluation with Probabilistic Guarantees

Information retrieval on Turkish texts

Contact Info

Product

Resources

About