Bitmap Algorithms for Counting Active Flows on High-Speed Links

Abstract

In this paper we present a family of algorithms that address the problem of counting the number of distinct header patterns (flows) seen on a high speed link. Such counting can be used to detect DoS attacks and port scans, and to solve measurement problems. The central difficulty is that count processing must be done within a packet arrival time (8 nsec at OC-768 speeds) and, hence, must require only a small number of memory references to limited, fast memory. A naive solution that maintains a hash table requires several Mbytes because the number of flows can be more than a million. By contrast, our new algorithms take very little memory and are fast. The reduction in memory is particularly important for applications that run multiple concurrent counting instances. For example, we used one of our new algorithms to replace the port scan detection component of the popular intrusion detection system Snort. Doing so reduced the memory usage on a ten minute trace from 50 Mbytes to 5.6 Mbytes while maintaining a 99.5% probability of alarming on a scan within 9 seconds of when the large-memory algorithm would alarm. By contrast, the best known prior algorithm (probabilistic counting) takes 4 times more memory on the port scan application and 8 times more memory on a measurement application. Our algorithms also lead to a reduction by a factor of seven in the total memory usage of a traffic analysis application from the CoralReef suite. Fundamentally, this is because our algorithms can be customized to take advantage of special features of applications such as a large number of instances that have very small counts or prior knowledge of the likely range of the count.

Count Me In: Viable Distributed Summary Statistics for Securing High-Speed Networks

“…Figure 6 shows a corresponding probability density for the three configurations. 9 We see that while using the summary statistics framework imposes overhead, it remains small for scan detection (0.4 percentage points more). The difference with the top-k script (1.6 percentage points more) is more noticeable due to the increased cost per observation that the more expensive maintenance of the probabilistic data structure entails.…”

“…To give just a few examples of research efforts presenting applications and/or corresponding data structures, the literature includes work on finding port scanners in backbones [24], efficiently counting the number of network flows in high-speed environments [16,9], detecting attacks against routers [1], computing real-time traffic summaries [15], or identifying elephant flows [8]. However, all of these efforts remain specific to their particular target application, while our work provides a framework on top of which one can implement such analyses.…”

Line speed accurate superspreader identification using dynamic error compensation

“…Identifying superspreaders provides very important information for many other applications, such as port scanner and Distributed Denial of Service (DDoS) attack detection, worm propagation measurement, and hot spots localization in peer-to-peer (p2p) and Content Delivery Networks (CDN) [1,2,3,4,5,6].…”

Adaptive sampling algorithm for detection of superpoints

“…In this algorithm, we also design an effective data streaming structure that can maintain the information of every flow and IP address through a high-speed link in the limited memory resource. The SDAS adopts multiple network measurement technologies, such as the data streaming structure [3][4][5] to record the flow, the sample and hold technique [6][7][8] to decrease the non-superpoints measurement and to improve the accuracy of the superpoints detection, the non-uniform sampling technique [9,10] to adjust the probability in the adaptive process. Experiment results and the mathematics analysis show that the SDAS is better than conventional algorithms in the adaptive function, resource consumption and measuring accuracy, so it can be used to detect the superpoints accurately and timely in the high speed links.…”