Operational requirements for scalable search systems

Chowdhury, Abdur; Pass, Greg

doi:10.1145/956863.956945

Cited by 28 publications

(11 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such algorithms could improve retrieval effectiveness by assisting in query disambiguation, making it easier to determine what information need is being expressed by a query at a given time. They could also assist research in search efficiency that takes into account query arrival-rates [3].…”

mentioning

confidence: 99%

Hourly analysis of a very large topically categorized web query log

Beitzel

Jensen

Chowdhury

et al. 2004

Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

237

158

View full text Add to dashboard Cite

We review a query log of hundreds of millions of queries that constitute the total query traffic for an entire week of a generalpurpose commercial web search service. Previously, query logs have been studied from a single, cumulative view. In contrast, our analysis shows changes in popularity and uniqueness of topically categorized queries across the hours of the day. We examine query traffic on an hourly basis by matching it against lists of queries that have been topically pre-categorized by human editors. This represents 13% of the query traffic. We show that query traffic from particular topical categories differs both from the query stream as a whole and from other categories. This analysis provides valuable insight for improving retrieval effectiveness and efficiency. It is also relevant to the development of enhanced query disambiguation, routing, and caching algorithms.

show abstract

mentioning

confidence: 99%

Hourly analysis of a very large topically categorized web query log

Beitzel

Jensen

Chowdhury

et al. 2004

Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Self Cite

237

158

View full text Add to dashboard Cite

show abstract

“…As a consequence of these tight performance constraints, search engines construct and maintain very large compute infrastructures for crawling the Web, indexing discovered pages, and processing user queries [3]. The scalability and efficiency of these infrastructures require careful performance optimizations in every major component of the search engine [2]. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.…”

Section: Descriptionmentioning

confidence: 99%

Scalability and efficiency challenges in commercial web search engines

Cambazoğlu

Baeza-Yates

2013

Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Commercial web search engines rely on very large compute infrastructures to be able to cope with the continuous growth of the Web and user bases. Achieving scalability and efficiency in such large-scale search engines requires making careful architectural design choices while devising algorithmic performance optimizations. Unfortunately, most details about the internal functioning of commercial web search engines remain undisclosed due to their financial value and the high level of competition in the search market. The main objective of this tutorial is to provide an overview of the fundamental scalability and efficiency challenges in commercial web search engines, bridging the existing gap between the industry and academia.

show abstract

“…The number of cluster replicas is determined based on a constraint on the peak sustainable throughput. In addition, the utilization of search nodes is taken into account (Chowdhury & Pass, 2003).…”

Section: Query Processing Overviewmentioning

confidence: 99%

A five-level static cache architecture for web search engines

Ozcan¹,

Altıngövde²,

Cambazoğlu³

et al. 2012

Information Processing & Management

View full text Add to dashboard Cite

Cataloged from PDF version of article.Caching is a crucial performance component of large-scale web search engines, as it greatly helps reducing average query response times and query processing workloads on backend search clusters. In this paper, we describe a multi-level static cache architecture that stores five different item types: query results, precomputed scores, posting lists, precomputed intersections of posting lists, and documents. Moreover, we propose a greedy heuristic to prioritize items for caching, based on gains computed by using items' past access frequencies, estimated computational costs, and storage overheads. This heuristic takes into account the inter-dependency between individual items when making its caching decisions, i.e., after a particular item is cached, gains of all items that are affected by this decision are updated. Our simulations under realistic assumptions reveal that the proposed heuristic performs better than dividing the entire cache space among particular item types at fixed proportions. (C) 2010 Elsevier Ltd. All rights reserved

show abstract

Operational requirements for scalable search systems

Cited by 28 publications

References 36 publications

Hourly analysis of a very large topically categorized web query log

Hourly analysis of a very large topically categorized web query log

Scalability and efficiency challenges in commercial web search engines

A five-level static cache architecture for web search engines

Contact Info

Product

Resources

About