Probabilistic counting algorithms for data base applications

Flajolet, Philippe; Martin, G. Nigel

doi:10.1016/0022-0000(85)90041-8

Cited by 1,030 publications

(906 citation statements)

References 4 publications

Supporting

Mentioning

871

Contrasting

Unclassified

Order By: Relevance

“…Vitter [26] shows how to sample in a small amount of space and linear time in a single pass. A related problem is computing the spectra (approximate number of distinct values) of a stream which can be achieved in lg n space [16,27]. Alon et al show that the first five moments can be approximated in lg n space while surprisingly all other (higher) moments require linear space [1].…”

Section: Introductionmentioning

confidence: 99%

Frequency Estimation of Internet Packet Streams with Limited Space

Demaine

López-Ortíz

Munro

2002

Lecture Notes in Computer Science

353

332

View full text Add to dashboard Cite

Abstract. We consider a router on the Internet analyzing the statistical properties of a TCP/IP packet stream. A fundamental difficulty with measuring traffic behavior on the Internet is that there is simply too much data to be recorded for later analysis, on the order of gigabytes a second. As a result, network routers can collect only relatively few statistics about the data. The central problem addressed here is to use the limited memory of routers to determine essential features of the network traffic stream. A particularly difficult and representative subproblem is to determine the top k categories to which the most packets belong, for a desired value of k and for a given notion of categorization such as the destination IP address.We present an algorithm that deterministically finds (in particular) all categories having a frequency above 1/(m + 1) using m counters, which we prove is best possible in the worst case. We also present a sampling-based algorithm for the case that packet categories follow an arbitrary distribution, but their order over time is permuted uniformly at random. Under this model, our algorithm identifies flows above a frequency threshold of roughly 1/ √ nm with high probability, where m is the number of counters and n is the number of packets observed. This guarantee is not far off from the ideal of identifying all flows (probability 1/n), and we prove that it is best possible up to a logarithmic factor. We show that the algorithm ranks the identified flows according to frequency within any desired constant factor of accuracy.

show abstract

Section: Introductionmentioning

confidence: 99%

Frequency Estimation of Internet Packet Streams with Limited Space

Demaine

López-Ortíz

Munro

2002

Lecture Notes in Computer Science

353

332

View full text Add to dashboard Cite

show abstract

“…In this paper we consider three kinds of synopses that each peer builds up and posts on a per-term basis, representing the global ids of documents (e.g., URLs or unique names of MP3 files) that a peer holds in its collection: Bloom filters [7], hash sketches [18], and min-wise permutations [9,10]. These techniques have been invented for approximate, low-error representation of sets or multisets.…”

Section: Contributionmentioning

confidence: 99%

“…Fundamentals for statistical synopses of (multi-)sets have a rich literature, including work on Bloom filters [7,17], hash sketches [18], and min-wise independent permutations [9,10]. We will overview these in Section 3.…”

Section: Related Workmentioning

confidence: 99%

“…Hash sketches were first proposed by Flajolet and Martin in [18], to probabilistically estimate the cardinality of a multiset S. [19] proposes a hash-based synopsis data structure and algorithms to support low-error and high-confidence estimates for general set expressions. Hash sketches rely on the existence of a pseudo-uniform hash function h() : S → [0, 1, .…”

Section: Min-wise Independent Permutations (Mips)mentioning

confidence: 99%

“…Then, the quantity R(S) = max d∈S ρ(d) provides an estimation of the value of log n. [18,16] present analyses and techniques to bound from above the error introduced, relying basically on using multiple bit vectors and averaging over their corresponding R positions.…”

Section: Min-wise Independent Permutations (Mips)mentioning

confidence: 99%

See 2 more Smart Citations

IQN Routing: Integrating Quality and Novelty in P2P Querying and Ranking

Michel

Bender

Triantafillou

et al. 2006

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. We consider a collaboration of peers autonomously crawling the Web. A pivotal issue when designing a peer-to-peer (P2P) Web search engine in this environment is query routing: selecting a small subset of (a potentially very large number of relevant) peers to contact to satisfy a keyword query. Existing approaches for query routing work well on disjoint data sets. However, naturally, the peers' data collections often highly overlap, as popular documents are highly crawled. Techniques for estimating the cardinality of the overlap between sets, designed for and incorporated into information retrieval engines are very much lacking. In this paper we present a comprehensive evaluation of appropriate overlap estimators, showing how they can be incorporated into an efficient, iterative approach to query routing, coined Integrated Quality Novelty (IQN). We propose to further enhance our approach using histograms, combining overlap estimation with the available score/ranking information. Finally, we conduct a performance evaluation in MINERVA, our prototype P2P Web search engine.

show abstract

Probabilistic analysis of adaptative sampling

Louchard

2000

Random Struct. Alg.

View full text Add to dashboard Cite

This paper analyzes the asymptotic properties of a classical algorithm: the adaptative sampling which solves the following problem; how to estimate the number M of distinct elements of a large collection of n data. Using tools such as the random tree and techniques such as Mellin transforms, combinatorial identities on Stirling numbers and Bessel functions, we analyze all moments and the asymptotic distribution function of the algorithm. © 1997 John Wiley & Sons, Inc. Random Struct. Alg., 10, 157–168 (1997)

show abstract

Probabilistic counting algorithms for data base applications

Cited by 1,030 publications

References 4 publications

Frequency Estimation of Internet Packet Streams with Limited Space

Frequency Estimation of Internet Packet Streams with Limited Space

IQN Routing: Integrating Quality and Novelty in P2P Querying and Ranking

Probabilistic analysis of adaptative sampling

Contact Info

Product

Resources

About