Bitmap algorithms for counting active flows on high speed links

Abstract

In this paper we present a family of algorithms that address the problem of counting the number of distinct header patterns (flows) seen on a high speed link. Such counting can be used to detect DoS attacks and port scans, and to solve measurement problems. The central difficulty is that count processing must be done within a packet arrival time (8 nsec at OC-768 speeds) and, hence, must require only a small number of memory references to limited, fast memory. A naive solution that maintains a hash table requires several Mbytes because the number of flows can be more than a million. By contrast, our new algorithms take very little memory and are fast. The reduction in memory is particularly important for applications that run multiple concurrent counting instances. For example, we used one of our new algorithms to replace the port scan detection component of the popular intrusion detection system Snort. Doing so reduced the memory usage on a ten minute trace from 50 Mbytes to 5.6 Mbytes while maintaining a 99.5% probability of alarming on a scan within 9 seconds of when the large-memory algorithm would alarm. By contrast, the best known prior algorithm (probabilistic counting) takes 4 times more memory on the port scan application and 8 times more memory on a measurement application. Our algorithms also lead to a reduction by a factor of seven in the total memory usage of a traffic analysis application from the CoralReef suite. Fundamentally, this is because our algorithms can be customized to take advantage of special features of applications such as a large number of instances that have very small counts or prior knowledge of the likely range of the count.

Why go logarithmic if we can go linear?

“…The second category [1,3,9,10,13,14,16,18,19,21,23,42] is a sketch-based approach that scans the entire traffic and hashes each element into a sketch. The sketch is used at query time to estimate the distinct count.…”

“…The bucketbased algorithms are either pure-bucket-based [3,42] or hybridbucket-based algorithms. Hybrid-bucket algorithms can be further classified into hybrid-bucket-sampling algorithms [21,4], and hybrid-bucket-logarithmic algorithms [14].…”

“…We report our experience with building a distinct count system at a major search engine, Ask.com, which abides by strict accuracy and efficiency constraints. This problem has received considerable attention (e.g., [13,14,16,21,26]) that is reviewed in Section 3. Given the strict accuracy and efficiency constraints of search engines, we discovered much merit in Linear Counting [42], an algorithm that is rarely used.…”

Window-based congestion control to improve TCP fairness and utilization for wide-bandwidth networks

“…According to the RED+ECN method [8], we can extract the queue length from the router. The number of flows in the router can be estimated with the Direct Bitmap [9]. Like EWA (Explicit Window Adaptation), conveying information from the router to senders may be achieved by inserting the value into the TCP header of each packet leaving the access router (for example, in the TCP option field) and can be conveyed using the RWND (Receiver Window) field of the IP header [9].…”

“…The number of flows in the router can be estimated with the Direct Bitmap [9]. Like EWA (Explicit Window Adaptation), conveying information from the router to senders may be achieved by inserting the value into the TCP header of each packet leaving the access router (for example, in the TCP option field) and can be conveyed using the RWND (Receiver Window) field of the IP header [9]. According to the information of the queue and the number of flows which is sent from a base station, each flow calculates its own window size that it must reach for fairness.…”