Uncovering Global Icebergs in Distributed Streams: Results and Implications

Huang, Guanyao; Lall, Ashwin; Chuah, Chen‐Nee; Xu, Jun

doi:10.1007/s10922-010-9186-5

Cited by 14 publications

(20 citation statements)

References 23 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As with most of previous work, Sample-Sketch [25] yields approximate answers due to its lossy nature. Sample-Sketch uses both sampling and sketching to identify global icebergs that are otherwise difficult to detect.…”

Section: A Abilene Trace Experimentsmentioning

confidence: 89%

“…We use 5-minute 11-node Abilene NetFlow traces (also used in [25] to study global iceberg detection using the SampleSketch scheme). The flow traces were originally collected with a 1 in 100 sampling rate.…”

Section: A Abilene Trace Experimentsmentioning

confidence: 99%

“…As one of the application instances, we use Pegasus to detect global icebergs, a set of flows with a common property contributing a significant amount of traffic across the entire network [25], [41], [42]. Example icebergs include bandwidthconsuming flows, distributed denial of service (DDoS) attacks, heavy-hitters in content distribution networks (CDNs) that denote "important" objects, anomalies, and worms.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Pegasus: Precision hunting for icebergs and anomalies in network flows

Gangam

Sharma²,

Fahmy

2013

2013 Proceedings IEEE INFOCOM

View full text Add to dashboard Cite

Abstract-Accurate online network monitoring is crucial for detecting attacks, faults, and anomalies, and determining traffic properties across the network. With high bandwidth links and consequently increasing traffic volumes, it is difficult to collect and analyze detailed flow records in an online manner. Traditional solutions that decouple data collection from analysis resort to sampling and sketching to handle large monitoring traffic volumes. We propose a new system, Pegasus, to leverage commercially available co-located compute and storage devices near routers and switches. Pegasus adaptively manages data transfers between monitors and aggregators based on traffic patterns and user queries.We use Pegasus to detect global icebergs or global heavy-hitters. Icebergs are flows with a common property that contribute a significant fraction of network traffic. For example, DDoS attack detection is an iceberg detection problem with a common destination IP. Other applications include identification of "top talkers," top destinations, and detection of worms and port scans. Experiments with Abilene traces, sFlow traces from an enterprise network, and deployment of Pegasus as a live monitoring service on PlanetLab show that our system is accurate and scales well with increasing traffic and number of monitors.

show abstract

Section: A Abilene Trace Experimentsmentioning

confidence: 89%

Section: A Abilene Trace Experimentsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Pegasus: Precision hunting for icebergs and anomalies in network flows

Gangam

Sharma²,

Fahmy

2013

2013 Proceedings IEEE INFOCOM

View full text Add to dashboard Cite

show abstract

“…For the case of detecting hierarchical heavy hitters, for example, combining sketches and sampling seems promising [13].…”

Section: Discussionmentioning

confidence: 99%

Resource/accuracy tradeoffs in software-defined measurement

Moshref

Govindan

2013

Proceedings of the Second ACM SIGCOMM Workshop on Hot Topics in Software Defined Networking

View full text Add to dashboard Cite

Previous work on network measurements have explored several primitives of increasing complexity for measurement tasks at individual nodes, ranging from counters to hashing to arbitrary code fragments. In an SDN network, these primitives may require significant bandwidth, memory and processing resources, and the resources dedicated to these can affect the accuracy of the eventual measurement. In this paper, we first qualitatively discuss the tradeoff space of resource usage versus accuracy for these different primitives as a function of the spatial and temporal measurement granularity, then quantify these tradeoffs in the context of hierarchical heavy hitter detection.

show abstract

“…Elephant flow identification approaches have been proposed [14], [15], which are useful for both traffic engineering and anomaly detection. Global iceberg detection has been addressed using sampling [16], sketches [17] as well as hybrid sampling/sketching solutions [18].…”

Section: Background and Related Workmentioning

confidence: 99%

Streaming Solutions for Fine-Grained Network Traffic Measurements and Analysis

Khan

Hosein

Ghiasi

et al. 2014

IEEE/ACM Trans. Networking

Self Cite

View full text Add to dashboard Cite

Abstract-Online network traffic measurements and analysis is critical for detecting and preventing any real-time anomalies in the network. We propose, implement, and evaluate an online, adaptive measurement platform, which utilizes real-time traffic analysis results to refine subsequent traffic measurements. Central to our solution is the concept of Multi-Resolution Tiling (MRT), a heuristic approach that performs sequential analysis of traffic data to zoom into traffic sub-regions of interest. However, MRT is sensitive to transient traffic spikes. In this paper, we propose three novel traffic streaming algorithms that overcome the limitations of MRT and can cater to varying degrees of computational and storage budgets, detection latency, and accuracy of query response. We evaluate our streaming algorithms on a highly parallel and programmable hardware as well as a traditional software based platforms. The algorithms demonstrate significant accuracy improvement over MRT in detecting anomalies consisting of synthetic hard-to-track elephant flows and global icebergs. Our proposed algorithms maintain the worst case complexities of the MRT, while incurring only a moderate increase in average resource utilization.

show abstract

Uncovering Global Icebergs in Distributed Streams: Results and Implications

Cited by 14 publications

References 23 publications

Pegasus: Precision hunting for icebergs and anomalies in network flows

Pegasus: Precision hunting for icebergs and anomalies in network flows

Resource/accuracy tradeoffs in software-defined measurement

Streaming Solutions for Fine-Grained Network Traffic Measurements and Analysis

Contact Info

Product

Resources

About