Technique for Estimating the Number of Active Flows in High-Speed Networks

Abstract

The online collection of coarse-grained traffic information, such as the total number of flows, is gaining in importance due to a wide range of applications, such as congestion control and network security. In this paper, we focus on an active queue management scheme called SRED since it estimates the number of active flows and uses the quantity to indicate the level of congestion. However, SRED has several limitations, such as instability in estimating the number of active flows and underestimation of active flows in the presence of non-responsive traffic. We present a Markov model to examine the capability of SRED in estimating the number of flows. We show how the SRED cache hit rate can be used to quantify the number of active flows. We then propose a modified SRED scheme, called hash-based two-level caching (HaTCh), which uses hashing and a two-level caching mechanism to accurately estimate the number of active flows under various workloads. Simulation results indicate that the proposed scheme provides a more accurate estimation of the number of active flows than SRED, stabilizes the estimation with respect to workload fluctuations, and prevents performance degradation by efficiently isolating non-responsive flows. Keywords: Flow estimation, Markov model, nonresponsive flows, SRED, HaTCh.Manuscript received July 25, 2007; revised Oct. 18, 2007. A preliminary version of the paper was presented at IEEE CDC, Dec. 2003 I. IntroductionMeasuring and monitoring traffic is an important but admittedly difficult problem, primarily because of the huge volume of traffic to be processed in high-speed networks. To accurately capture the characteristics of network traffic, measuring devices have to maintain per-flow information. However, the complexity of these operations has been the main obstacle for deploying such devices in high-speed networks. To address this problem, various techniques have been proposed. For example, a sampling technique was standardized by the IETF Internet Protocol Flow Information Export (IPFIX) working group [1], and a variation focused on implementation issues was investigated in [2]. Recently, a variation of a hashing scheme called a space-code bloom filter (SCBF) [3] was proposed to avoid the overhead of per-flow maintenance. An SCBF collects network traffic on the arrival of each packet and periodically stores it in permanent storage devices. The required network information can then be obtained offline. Therefore, SCBF can support fine-grained traffic information, which is essential for network management, planning, accounting, and billing.On the other hand, obtaining coarse-grained traffic information, such as the total number of active flows, online is important for congestion control and network security [4]- [7]. In this context, flow counting techniques, called direct bitmap and multiresolution bitmap, have been proposed by Estan and others in [7]. These techniques are based on a bitmap data structure, in which each source is hashed into a bit, and a bit is marked when the source is...

A dynamic quarantine scheme for controlling unresponsive TCP sessions

“…Recently, we have proposed a flow estimation scheme, called HaTCh [43], that extended SRED's single cache model [31] into a hashing and two-level caching scheme for a more robust estimation of the number of active flows (N ) under various workloads. On each packet arrival, HaTCh computes the "hit" frequency, f (t), which is obtained using a first order auto-regressive function, and the inverse of the hit frequency (1/f (t )) is used as the estimate of the number of active flows (N ).…”

“…2. It was shown in [43] that HaTCh provides a more robust estimate of the number of active flows (N ) than SRED.…”

“…Third, HaTCh's estimation error in the number of active flows should be considered. Although HaTCh is a more robust estimator than SRED, its estimate also exhibits oscillation [43]. Therefore, we use certain multiples of C/N , where C is the link bandwidth, as the detection threshold, θ det , in identifying the unresponsive flows to reduce the false positives.…”

“…2.1. The size of the quarantine memory of HaDQ was configured equal to HaTCh's L 1 cache targeting a maximum of 100 unresponsive flows (note that this is less conservative than the "maximum aggressive flow" limit in [43]). …”

“…Although the proposed scheme can be used for controlling all types of high bandwidth flows, we restrict our discussion to only malicious TCP flows here. We propose a comprehensive solution that involves sampling, detection, and punitive measures for such flows, The proposed scheme, called HaDQ (HaTCh-based Dynamic Quarantine), builds upon the recently proposed HaTCh technique [43], which can provide an accurate and robust estimation of the number of active flows compared to the single cache-based SRED mechanism [31]. To estimate the number of active TCP flows through a packet memory in an Internet router, the HaTCh scheme uses a small L 1 cache that primarily captures the aggressive flows and a larger L 2 cache that monitors the normal flows for estimating the "hit" frequency and subsequently the number of flows.…”