A Large-Scale Network Data Analysis via Sparse and Low Rank Reconstruction

Lu, Liang; Huang, Zheng-Hai; Ambusaidi, Mohammed A.; Gou, Kui-Xiang

doi:10.1155/2014/323764

“…For example, traffic anomalies may be caused by problems ranging from security threats (e.g., Distributed Denial of Service (DDoS) attacks and network worms) to unusual traffic events (e.g., flash crowds), to vendor implementation bugs, and to network misconfigurations. Such anomalies are typically not known a priori and are sparse [11,25].…”

Section: Lens Decomposition Frameworkmentioning

confidence: 99%

Robust Network Compressive Sensing

Xue¹,

Chen²,

Lyu³

et al. 2022

SpringerBriefs in Computer Science

0

View full text Add to dashboard Cite

Networks are constantly generating an enormous amount of rich diverse information. Such information creates exciting opportunities for network analytics. However, a major challenge to enable effective network analytics is the presence of missing data, measurement errors, and anomalies. Despite significant work in network analytics, fundamental issues remain: (i) the existing works do not explicitly account for anomalies or measurement noise, and incur serious performance degradation under significant noise or anomalies, and (ii) they assume network matrices have low-rank structure, which may not hold in reality.To address these issues, in this paper we develop LENS decomposition, a novel technique to accurately decompose a network matrix into a low-rank matrix, a sparse anomaly matrix, an error matrix, and a small noise matrix. LENS has the following nice properties: (i) it is general: it can effectively support matrices with or without anomalies, and having low-rank or not, (ii) its parameters are self tuned so that it can adapt to different types of data, (iii) it is accurate by incorporating domain knowledge, such as temporal locality, spatial locality, and initial estimate (e.g., obtained from models), (iv) it is versatile and can support many applications including missing value interpolation, prediction, and anomaly detection. We apply LENS to a wide range of network matrices from 3G, WiFi, mesh, sensor networks, and the Internet. Our results show that LENS significantly out-performs state-of-the-art compressive sensing schemes.

show abstract

“…Much research has focused on the network traffic in the flow-level network. Therefore, a novel approach was proposed to reveal the abnormal patterns by dealing with the packet-level network data based on the latest method from compressed sensing [17].…”

Section: Anomaly-based Detectionmentioning

confidence: 99%

Big Data in Intrusion Detection Systems and Intrusion Prevention Systems

Wang¹

2017

JCN

View full text Add to dashboard Cite

This paper introduces network attacks, intrusion detection systems, intrusion prevention systems, and intrusion detection methods including signature-based detection and anomaly-based detection. Intrusion detection/prevention system (ID/PS) methods are compared. Some data mining and machine learning methods and their applications in intrusion detection are introduced. Big data in intrusion detection systems and Big Data analytics for huge volume of data, heterogeneous features, and real-time stream processing are presented. Challenges of intrusion detection systems and challenges posed by stream processing of big data in the systems are also discussed.

show abstract

“…For example, traffic anomalies may be caused by problems ranging from security threats (e.g., Distributed Denial of Service (DDoS) attacks and network worms) to unusual traffic events (e.g., flash crowds), to vendor implementation bugs, and to network misconfigurations. Such anomalies are typically not known a priori and are sparse [11,25]. Note that there can be systematic effects that are only sparse after some transformation (e.g., wavelet transform).…”

Section: Lens Decomposition Frameworkmentioning

confidence: 99%

Robust network compressive sensing

Chen

¹

,

Qiu

²

,

Zhang

³

et al. 2014

Proceedings of the 20th Annual International Conference on Mobile Computing and Networking

View full text Add to dashboard Cite

Networks are constantly generating an enormous amount of rich diverse information. Such information creates exciting opportunities for network analytics. However, a major challenge to enable effective network analytics is the presence of missing data, measurement errors, and anomalies. Despite significant work in network analytics, fundamental issues remain: (i) the existing works do not explicitly account for anomalies or measurement noise, and incur serious performance degradation under significant noise or anomalies, and (ii) they assume network matrices have low-rank structure, which may not hold in reality.To address these issues, in this paper we develop LENS decomposition, a novel technique to accurately decompose a network matrix into a low-rank matrix, a sparse anomaly matrix, an error matrix, and a small noise matrix. LENS has the following nice properties: (i) it is general: it can effectively support matrices with or without anomalies, and having low-rank or not, (ii) its parameters are self tuned so that it can adapt to different types of data, (iii) it is accurate by incorporating domain knowledge, such as temporal locality, spatial locality, and initial estimate (e.g., obtained from models), (iv) it is versatile and can support many applications including missing value interpolation, prediction, and anomaly detection. We apply LENS to a wide range of network matrices from 3G, WiFi, mesh, sensor networks, and the Internet. Our results show that LENS significantly out-performs state-of-the-art compressive sensing schemes.

show abstract

A Large-Scale Network Data Analysis via Sparse and Low Rank Reconstruction

Cited by 5 publications

References 24 publications

Robust Network Compressive Sensing

Robust Network Compressive Sensing

Big Data in Intrusion Detection Systems and Intrusion Prevention Systems

Robust network compressive sensing

Contact Info

Product

Resources

About