A fast sketch for aggregate queries over high-speed network traffic

Abstract

There have been security problems and network failures that are hard to resolve, for example, botnets, polymorphic worm/virus, DDoS, etc. To address them, we need to monitor the traffic dynamics and have a network-wide view about them, and more importantly, be able to detect attacks and failures in a timely manner. Due to the rapid increase in the traffic volume, it is often infeasible to monitor every individual flow in the backbone network due to space and time constraints. Instead, we are often required to aggregate packets into a small number of flows and develop the detection methods with aggregated flows, namely aggregate queries. Although it enables ISPs to detect network problems in a timely manner, the flow aggregation cannot preserve certain critical information in network traffic, e.g., IP addresses, port numbers, etc. Due to such missing information, it becomes very difficult (or often infeasible) for ISPs to identify the sources of network attacks or the causes of traffic anomalies, which are important to resolve the network problems effectively. In this paper, we propose an efficient data structure, namely the fast sketch, which can both aggregate packets into a small number of flows, and further enable ISPs to identify the anomalous keys (IP addresses, port numbers, etc.), with small space and time. With it, the number of aggregated flows can achieve the lower bound of the heavychange detection, i.e., Ω(k log(n/k)), where n is the range of flow keys and k is an upper bound of the number of anomalous keys. In addition, our sketch combines both the combinatorial group testing and the quotient technique to identify anomalous keys, which can guarantee a sub-linear running time. We expect our work will improve the practice for real-time traffic monitoring in a high-speed networked system.

Identifying High-Cardinality Hosts from Network-Wide Traffic Measurements

“…Both sketches utilize the position information to identify interested hosts in the network. A similar recovery method has also been proposed for traffic anomaly detection with the Principle Component Analysis (PCA) [15] and the aggregate queries in high-speed network [16].…”

“…In order to provide a small failure probability δ, this algorithm has to be repeated log(1/δ) times independently. If this algorithm is used in reversible sketches [12], [14], [16], the space and the running time would be very high for the detection of high-cardinality hosts. Therefore, in this paper we propose a more efficient data structure than trivial solutions, which considers the errors in cardinality estimation.…”

A Hybrid Approach Combining Rule-Based and Anomaly-Based Detection Against DDoS Attacks

“…We may categorize them into four main approaches: traceback-based [1][2][3][4][5], rule-based [6][7][8][9][10][11][12][13][14][15][16], protocolbased [17][18][19][20][21][22][23][24][25][26][27] and anomaly-based [28][29][30]. Traceback-based methods make the victim to identify the attack source as well as attack paths once the attack has been encountered.…”

“…They are network/transport-level [18][19][20][21][22][23] and application-level flooding [24][25][26][27]. Network/transport-level flooding has been launched attacks to consume the victims' resource by exploiting the bugs and the weakness of IP, TCP, UDP and ICMP protocols.…”

RES: A Resource-Efficient System for Monitoring Flow Distribution

“…People are using and relying on a large variety of services built on the top of the Internet, such as web browsing, online banking, shopping, entertainment, VoIP, Video on demand, auction, social networks, etc [1]. However, some network attacks including DDOS, information phishing and email spamming are pervasive in the Internet, and often cause great financial loss [2].…”