Bitmap algorithms for counting active flows on high speed links

Abstract

In this paper we present a family of algorithms that address the problem of counting the number of distinct header patterns (flows) seen on a high speed link. Such counting can be used to detect DoS attacks and port scans, and to solve measurement problems. The central difficulty is that count processing must be done within a packet arrival time (8 nsec at OC-768 speeds) and, hence, must require only a small number of memory references to limited, fast memory. A naive solution that maintains a hash table requires several Mbytes because the number of flows can be more than a million. By contrast, our new algorithms take very little memory and are fast. The reduction in memory is particularly important for applications that run multiple concurrent counting instances. For example, we used one of our new algorithms to replace the port scan detection component of the popular intrusion detection system Snort. Doing so reduced the memory usage on a ten minute trace from 50 Mbytes to 5.6 Mbytes while maintaining a 99.5% probability of alarming on a scan within 9 seconds of when the large-memory algorithm would alarm. By contrast, the best known prior algorithm (probabilistic counting) takes 4 times more memory on the port scan application and 8 times more memory on a measurement application. Our algorithms also lead to a reduction by a factor of seven in the total memory usage of a traffic analysis application from the CoralReef suite. Fundamentally, this is because our algorithms can be customized to take advantage of special features of applications such as a large number of instances that have very small counts or prior knowledge of the likely range of the count.

Robust network monitoring in the presence of non-cooperative traffic queries

“…In order to extract the features with minimum overhead, we implement the multi-resolution bitmap algorithms proposed in [13]. The advantage of the multi-resolution bitmaps is that they bound the number of memory accesses per packet as compared to classical hash tables and they can handle a large number of items with good accuracy and small memory footprint.…”

A New Algorithm for Long Flows Statistics—MGCBF

“…This algorithm applied maximum likelihood estimation (MLE) and mean value estimation (MVE) to estimate the length of every flow. C. Estan, G. Varghese and M. Fisk [10] described a bitmap algorithm to take count of the flows with different length with little storage resource. But all those algorithms and methods do not maintain the detail information for flows, which can satisfy the needs of special applications (i.e.…”

A robust system for accurate real-time summaries of internet traffic

“…This problem is algorithmically equivalent to the problem of finding superspreaders which has been addressed by Venkataraman et al [29] and they have independently proposed techniques equivalent to our flow sample and hold and Bloom filter tuple set counting. Efficient bitmap algorithms for flow counting have been proposed [15]. Our system directly uses some of them while improving on others.…”

“…The "global counters" that are part of the reports do not pose significant issues: for the bytes and packets we use simple counters, and for counting distinct addresses, ports, and flows we use multiresolution bitmaps [15]. Since these have low and constant CPU and memory usage and well understood accuracy, the rest of the paper will focus on the hog reports.…”

“…Start with an empty bitmap, set the bit in the bitmap corresponding to the hash value of the flow ID of each packet, and at the end of the interval estimate the number of active flows based on the number of bits set. This algorithm, called linear counting [31] or direct bitmap [15], provides accurate estimates but its memory requirements scale almost linearly with the maximum number of active flows. The size of the bitmap also depends on the required accuracy of the estimate.…”