Frequency Estimation of Internet Packet Streams with Limited Space

Abstract

Abstract. We consider a router on the Internet analyzing the statistical properties of a TCP/IP packet stream. A fundamental difficulty with measuring traffic behavior on the Internet is that there is simply too much data to be recorded for later analysis, on the order of gigabytes a second. As a result, network routers can collect only relatively few statistics about the data. The central problem addressed here is to use the limited memory of routers to determine essential features of the network traffic stream. A particularly difficult and representative subproblem is to determine the top k categories to which the most packets belong, for a desired value of k and for a given notion of categorization such as the destination IP address.We present an algorithm that deterministically finds (in particular) all categories having a frequency above 1/(m + 1) using m counters, which we prove is best possible in the worst case. We also present a sampling-based algorithm for the case that packet categories follow an arbitrary distribution, but their order over time is permuted uniformly at random. Under this model, our algorithm identifies flows above a frequency threshold of roughly 1/ √ nm with high probability, where m is the number of counters and n is the number of packets observed. This guarantee is not far off from the ideal of identifying all flows (probability 1/n), and we prove that it is best possible up to a logarithmic factor. We show that the algorithm ranks the identified flows according to frequency within any desired constant factor of accuracy.

Approximate Range Mode and Range Median Queries

“…The idea presented in [3] leads to an algorithm that maintains a counter for each element in the search tree: The counters are stored in a doubly linked list of groups, each group is a collection of elements with equal counter value. Groups are ordered according to their counter values.…”

Differentially Private Continual Monitoring of Heavy Hitters from Distributed Streams

“…They designed a deterministic algorithm that reads a stream of W items and at the end gives the count of every item in the stream with relative error λ (i.e., additive error at most λW ); the algorithm only uses O( 1 λ ) words of memory. The MG algorithm was rediscovered several times [5,13]. Using the MG algorithm concurrently on overlapping blocks of different sizes, Arasu and Manku [1] gave a deterministic algorithm that continually estimate the count of every item with relative error λ with respect to the current window; the query and the update time is O( 1 λ log 1 λ ), while O( 1 λ log 2 1 λ ) words of memory is required.…”

A Model of Wireless Sensor Networks Using Opportunistic Routing in Logistic Harbor Scenarios

“…The neighbor management component is responsible for receiving beacons and validating these beacons to avoid loops or duplications. After that, the information in a valid beacon is used to update the neighbor table with a management policy: the insert or update rate is normally equal to the beacon rate, but the evict rate is defined by a timer using the frequency algorithm [7] to count the number of received beacons from each neighbor and use it as a quality indicator. This can help to classify the neighbor nodes: only the high-quality nodes, from which the current node receives many update beacons, are kept in the neighbor table.…”