Fast and Accurate Traffic Measurement With Hierarchical Filtering

Wang, Haibo; Xu, Hongli; Huang, Liusheng; Zhai, Yutong

doi:10.1109/tpds.2020.2991007

Cited by 8 publications

(3 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…the supported queries. Data summaries-Sketches: Another approach for computing network summaries are sketches, e.g., [73], [34], [74] as well as systems that utilize sketches for network monitoring and debugging [75], [25], [35], [76], [77]. The capabilities of sketches include counting, top-K, HH, as well as HHH.…”

Section: State Of the Artmentioning

confidence: 99%

Exploring Network-Wide Flow Data With Flowyager

Saidi

Maghsoudlou

Foucard

et al. 2020

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

Many network operations, ranging from attack investigation and mitigation to traffic management, require answering network-wide flow queries in seconds. Although flow records are collected at each router, using available traffic capture utilities, querying the resulting datasets from hundreds of routers across sites and over time, remains a significant challenge due to the sheer traffic volume and distributed nature of flow records. In this paper, we investigate how to improve the response time for a priori unknown network-wide queries. We present Flowyager, a system that is built on top of existing traffic capture utilities. Flowyager generates and analyzes tree data structures, that we call Flowtrees, which are succinct summaries of the raw flow data available by capture utilities. Flowtrees are self-adjusted data structures that drastically reduce space and transfer requirements, by 75% to 95%, compared to raw flow records. Flowyager manages the storage and transfers of Flowtrees, supports Flowtree operators, and provides a structured query language for answering flow queries across sites and time periods. By deploying a Flowyager prototype at both a large Internet Exchange Point and a Tier-1 Internet Service Provider, we showcase its capabilities for networks with hundreds of router interfaces. Our results show that the query response time can be reduced by an order of magnitude when compared with alternative data analytics platforms. Thus, Flowyager enables interactive network-wide queries and offers unprecedented drill-down capabilities to, e.g., identify DDoS culprits, pinpoint the involved sites, and determine the length of the attack. 1 NetFlow is a Cisco trademark, so other vendors market the NetFlow support with other names, e.g., Juniper Networks use the trademark Jflow or cflowd. 2 NetFlow and IPFIX capabilities are available in router series, e.g., Cisco IOS-XR, IOS and Catalyst router [

show abstract

Section: State Of the Artmentioning

confidence: 99%

Exploring Network-Wide Flow Data With Flowyager

Saidi

Maghsoudlou

Foucard

et al. 2020

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

show abstract

“…For example, Yahoo's Hadoop, PIG [15], and Microsoft's Cosmos [16] use multiplicity queries to assist the analysis of enormous data sets, such as web crawls, search logs, and click streams. Besides, many network measurement tasks [17], [18], including delay, burst detection, flow size estimation, flow distribution, and heavy hitter detection [19], [20], [21], [22], [23], [24], also often use multiplicity query.…”

Section: Introductionmentioning

confidence: 99%

A Shifting Filter Framework for Dynamic Set Queries

Luo

Guo

et al. 2023

IEEE/ACM Trans. Networking

View full text Add to dashboard Cite

Set query is a fundamental problem in computer systems. Plenty of applications rely on the query results of membership, association, and multiplicity. A traditional method that addresses such a fundamental problem is derived from Bloom filter. However, such methods may fail to support element deletion, require additional filters or apriori knowledge, making them unamenable to a high-performance implementation for dynamic set representation and query. In this paper, we envision a novel sketch framework that is multi-functional, non-parametric, space efficient, and deletable. As far as we know, none of the existing designs can guarantee such features simultaneously. To this end, we present a general shifting framework to represent auxiliary information (such as multiplicity, association) with the offset. Thereafter, we specify such design philosophy for a hash table horizontally at the slot level, as well as vertically at the bucket level. Theoretical and experimental results jointly demonstrate that our design works exceptionally well with three types of set queries under small memory.

show abstract

“…Traffic measurement provides valuable information for network security and network management, such as traffic accounting, load balancing, and anomaly detection [1][2][3][4][5], where measuring cardinality is an important task of traffic measurement. Cardinality measurement is a prerequisite for detecting supernodes, which have been paid extensive attention to by both academic and industrial organizations, despite many efforts in detecting supernodes over recent decades.…”

Section: Introductionmentioning

confidence: 99%

An Efficient Method for Detecting Supernodes Using Reversible Summary Data Structures in the Distributed Monitoring Systems

Zhou

Jin

2022

Security and Communication Networks

View full text Add to dashboard Cite

Supernode detection has many applications in detecting network attacks, assisting resource allocation, etc. As 5G/IoT networks constantly grow, big network traffic brings a great challenge to collect massive traffic data in compact and real-time way. Previous works focus on detecting supernodes in a measurement point, while only a few works consider it in the distributed monitoring system. Moreover, they are not able to measure two types of node cardinalities simultaneously and reconstruct labels of supernodes efficiently due to large calculation and memory cost. To address these problems, we propose a novel reversible and distributed traffic summarization called RDS to simultaneously measure source and destination cardinalities for detecting supernodes in the distributed monitoring system. The basic idea of our approach is that each monitor generates a summary data structure using the coming packets and sends the summary data structure to the controller; then, the controller aggregates the received summary data structures, estimates node cardinalities, and reconstructs labels of supernodes according to the aggregated summary data structure. The experimental results based on real network traffic demonstrate that the proposed approach can detect up to 96% supernodes with a low memory requirement in comparison with state-of-the-art approaches.

show abstract

Fast and Accurate Traffic Measurement With Hierarchical Filtering

Cited by 8 publications

References 45 publications

Exploring Network-Wide Flow Data With Flowyager

Exploring Network-Wide Flow Data With Flowyager

A Shifting Filter Framework for Dynamic Set Queries

An Efficient Method for Detecting Supernodes Using Reversible Summary Data Structures in the Distributed Monitoring Systems

Contact Info

Product

Resources

About