Reversible sketches for efficient and accurate change detection over network data streams

Schweller, Robert T.; Gupta, Ashish; Parsons, Elliot; Chen, Yan

doi:10.1145/1028788.1028814

Cited by 135 publications

(78 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The deltoid expands k-ary sketch with multiple counters for each bucket in the hash tables. The reverse hashing method further improves sketchbased change detection, which is more efficient and can infer the keys of culprit flows [22]. The work done by Xu et al [27] is closest to our work, in which traffic behavior profiling is conducted in real-time.…”

Section: Related Workmentioning

confidence: 76%

“…Duffield et al [6] showed that even packet sampling is not scalable, especially after aggregation. There can be up to 2 64 flows defined by only considering source and destination IP addresses [22]. Recently, Cormode et al [4]'s deltoids and Krishnamurthy et al [11]'s k-ary sketch methods have been proposed for heavy change detection in high speed traffic.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Online detection of network traffic anomalies using behavioral distance

Sengar¹,

Wang

et al. 2009

2009 17th International Workshop on Quality of Service

View full text Add to dashboard Cite

Abstract-While network-wide anomaly analysis has been well studied, the on-line detection of network traffic anomalies at a vantage point inside the Internet still poses quite a challenge to network administrators. In this paper, we develop a behavioral distance based anomaly detection mechanism with the capability of performing on-line traffic analysis. To construct accurate online traffic profiles, we introduce horizontal and vertical distance metrics between various traffic features (i.e., packet header fields) in the traffic data streams. The significant advantages of the proposed approach lie in four aspects: (1) it is efficient and simple enough to process on-line traffic data; (2) it facilitates protocol behavioral analysis without maintaining per-flow state; (3) it is scalable to high speed traffic links because of the aggregation, and (4) using various combinations of packet features and measuring distances between them, it is capable for accurate on-line anomaly detection. We validate the efficacy of our proposed detection system by using network traffic traces collected at Abilene and MAWI high-speed links.

show abstract

Section: Related Workmentioning

confidence: 76%

Section: Related Workmentioning

confidence: 99%

Online detection of network traffic anomalies using behavioral distance

Sengar¹,

Wang

et al. 2009

2009 17th International Workshop on Quality of Service

View full text Add to dashboard Cite

show abstract

“…Thus, in this work, to identify the culprits, the original keys were kept temporarily. For real-world applications that must avoid high memory consumption to maintain the original keys, reversible sketches [23] can be efficiently applied. As the focus of our work is detection performance, we did not utilize or implement the reversible sketches.…”

Section: Discussionmentioning

confidence: 99%

Detecting Anomalies in Massive Traffic Streams Based on S-Transform Analysis of Summarized Traffic Entropies

Pukkawanna

Hazeyama

Kadobayashi

et al. 2015

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYDetecting traffic anomalies is an indispensable component of overall security architecture. As Internet and traffic data with more sophisticated attacks grow exponentially, preserving security with signaturebased traffic analyzers or analyzers that do not support massive traffic are not sufficient. In this paper, we propose a novel method based on combined sketch technique and S-transform analysis for detecting anomalies in massive traffic streams. The method does not require any prior knowledge such as attack patterns and models representing normal traffic behavior. To detect anomalies, we summarize the entropy of traffic data over time and maintain the summarized data in sketches. The entropy fluctuation of the traffic data aggregated to the same bucket is observed by S-transform to detect spectral changes referred to as anomalies in this work. We evaluated the performance of the method with real-world backbone traffic collected at the United States and Japan transit link in terms of both accuracy and false positive rates. We also explored the method parameters' influence on detection performance. Furthermore, we compared the performance of our method to S-transform-based and Wavelet-based methods. The results demonstrated that our method was capable of detecting anomalies and overcame both methods. We also found that our method was not sensitive to its parameter settings.

show abstract

“…To meet the requirements above, we propose a new paradigm called DoS resilient High-speed Flow-level INtrusion Detection, HiFIND [18] leveraging recent work on data streaming computation and in particular, sketches [19,20]. Sketches are a kind of compact data streaming data structure which record traffic for given keys and are capable of reporting heavy traffic keys.…”

Section: Fifth Separating Anomalies From Intrusions For False Positimentioning

confidence: 99%

“…Although proposed in [19,20], sketches have not been applied to building IDSes for the following reasons:…”

Section: Fifth Separating Anomalies From Intrusions For False Positimentioning

confidence: 99%

HiFIND: A high-speed flow-level intrusion detection approach with DoS resiliency

Gao

Chen

2010

Computer Networks

Self Cite

View full text Add to dashboard Cite

a b s t r a c tGlobal-scale attacks like worms and botnets are increasing in frequency, severity and sophistication, making it critical to detect outbursts at routers/gateways instead of end hosts. In this paper, leveraging data streaming techniques such as the reversible sketch, we design HiFIND, a High-speed Flow-level Intrusion Detection system. In contrast to existing intrusion detection systems, HiFIND: (i) is scalable to flow-level detection on high-speed networks; (ii) is DoS resilient; (iii) can distinguish SYN flooding and various port scans (mostly for worm propagation) for effective mitigation; (iv) enables aggregate detection over multiple routers/gateways; and (v) separates anomalies to limit false positives in detection. Both theoretical analysis and evaluation with several router traces show that HiFIND achieves these properties. To the best of our knowledge, HiFIND is the first online DoS resilient flow-level intrusion detection system for high-speed networks (e.g. OC192), even for the worst-case traffic of 40-byte-packet streams with each packet forming a flow.

show abstract

Reversible sketches for efficient and accurate change detection over network data streams

Cited by 135 publications

References 23 publications

Online detection of network traffic anomalies using behavioral distance

Online detection of network traffic anomalies using behavioral distance

Detecting Anomalies in Massive Traffic Streams Based on S-Transform Analysis of Summarized Traffic Entropies

HiFIND: A high-speed flow-level intrusion detection approach with DoS resiliency

Contact Info

Product

Resources

About