Network security and anomaly detection with Big-DAMA, a big data analytics framework

Casas, Pedro; Soro, Francesca; Vanerio, Juan; Settanni, Giuseppe; D’Alconzo, Alessandro

doi:10.1109/cloudnet.2017.8071525

Cited by 28 publications

(41 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Network traffic monitoring and analysis is considered from the point of view of big data analytics in Reference . The Big‐DAMA framework is presented, which can store and process both structured and unstructured data from heterogeneous sources, with both stream and batch processing capabilities.…”

Section: Related Workmentioning

confidence: 99%

“…SPE's have been applied to sensor networks, the analysis of TCP flows, network intrusion detection . Multiple, diverse strategies for traffic anomaly detection have been proposed: machine learning, expert systems, rule mining, NFV technology, besides the strategy implemented in the present work which is the usage of entropy and PCA to detect traffic anomalies by monitoring the network traffic as a whole . It is possible to say that the major contribution of the present work is to employ an SPE extended with the operators to compute traffic entropy and PCA of the network as a whole in real time.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

BackStreamDB: A stream processing engine for backbone traffic monitoring with anomaly detection

Duarte

Hara

Torres

et al. 2020

Security and Privacy

View full text Add to dashboard Cite

BackStreamDB is distributed traffic monitoring system based on a stream processing engine (SPE) designed to monitor the traffic of wide area backbones.BackStreamDB provides arbitrary metrics about the traffic in real time, taking into account the backbone as a whole. The system was developed for and successfully deployed on the Brazilian National Academic Network (RNP). In this work, we describe the functionality for the detection of traffic anomalies. A large number of Internet attacks are continuously reported, and several types of attacks result in anomalous traffic. In the proposed strategy for anomaly detection, the traffic is sampled by monitors that are distributed across the backbone, which are accessed and processed by the SPE. BackStreamDB was extended with stream processing modules for computing traffic entropy and principal component analysis, which are the employed to detect traffic anomalies. Experimental results are reported which were obtained to validate the effectiveness of the proposed strategy for different types of attacks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

BackStreamDB: A stream processing engine for backbone traffic monitoring with anomaly detection

Duarte

Hara

Torres

et al. 2020

Security and Privacy

View full text Add to dashboard Cite

show abstract

“…We refer the interested reader to [1] for a detailed survey on the different machine-learning techniques commonly applied to network-traffic analysis. There are multiple recent papers on the application of machine-learning models to networksecurity and anomaly-detection problems [3], [7]- [9]. In [3], we analyze and benchmark big-data-analytics frameworks for large-scale network-traffic monitoring and analysis.…”

Section: State Of the Artmentioning

confidence: 99%

ADAM & RAL: Adaptive Memory Learning and Reinforcement Active Learning for Network Monitoring

Wassermann

Cuvelier

Mulinka

et al. 2019

2019 15th International Conference on Network and Service Management (CNSM)

Self Cite

View full text Add to dashboard Cite

Network-traffic data commonly arrives in the form of fast data streams; online network-monitoring systems continuously analyze these kinds of streams, sequentially collecting measurements over time. Continuous and dynamic learning is an effective learning strategy when operating in these fast and dynamic environments, where concept drifts constantly occur. In this paper, we propose different approaches for stream-based machine learning, able to analyze network-traffic streams on the fly, using supervised learning techniques. We address two major challenges associated to stream-based machine learning and online network monitoring: (i) how to dynamically learn from and adapt to non-stationary data and patterns changing over time, and (ii) how to deal with the limited availability of ground truth or labeled data to continuously tune a supervised learning model. We introduce ADAM & RAL, two stream-based machine-learning approaches to tackle these challenges. ADAM implements multiple stream-based machine-learning models and relies on an adaptive memory strategy to dynamically adapt the size of the system's learning memory to the most recent data distribution, triggering new learning steps when concept drifts are detected. RAL implements a stream-based active-learning strategy to reduce the amount of labeled data needed for streambased learning, dynamically deciding on the most informative samples to integrate into the continuous learning scheme. Using a reinforcement learning loop, RAL improves prediction performance by additionally learning from the goodness of its previous sample-selection decisions. We focus on a particularly challenging problem in network monitoring: continuously tuning detection models able to recognize network attacks over time. By continuously learning from and detecting concept drifts within real network measurements, we show that ADAM & RAL can continuously achieve high detection accuracy and limit the amount of training data needed to detect attacks over dynamic network data streams.

show abstract

“…Casas et al [15] have used Apache Spark Streaming (RDD version) to detect anomalies and have compared its performance with other frameworks. Frameworks that are specifically built for anomaly detection have shown good performance; better than that of Spark Streaming.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Both Casas et al [15] and Callegari et al [16] used the MAWILab dataset. Since 2001, 15 min network traffic traces are captured on a backbone link between Japan and the US.…”

Section: Dataset Descriptionmentioning

confidence: 99%

Comparative Study between Big Data Analysis Techniques in Intrusion Detection

Hafsa

Farah

2018

BDCC

View full text Add to dashboard Cite

Cybersecurity ventures expect that cyber-attack damage costs will rise to $11.5 billion in 2019 and that a business will fall victim to a cyber-attack every 14 seconds. Notice here that the time frame for such an event is seconds. With petabytes of data generated each day, this is a challenging task for traditional intrusion detection systems (IDSs). Protecting sensitive information is a major concern for both businesses and governments. Therefore, the need for a real-time, large-scale and effective IDS is a must. In this work, we present a cloud-based, fault tolerant, scalable and distributed IDS that uses Apache Spark Structured Streaming and its Machine Learning library (MLlib) to detect intrusions in real-time. To demonstrate the efficacy and effectivity of this system, we implement the proposed system within Microsoft Azure Cloud, as it provides both processing power and storage capabilities. A decision tree algorithm is used to predict the nature of incoming data. For this task, the use of the MAWILab dataset as a data source will give better insights about the system capabilities against cyber-attacks. The experimental results showed a 99.95% accuracy and more than 55,175 events per second were processed by the proposed system on a small cluster.

show abstract

Network security and anomaly detection with Big-DAMA, a big data analytics framework

Cited by 28 publications

References 20 publications

BackStreamDB: A stream processing engine for backbone traffic monitoring with anomaly detection

BackStreamDB: A stream processing engine for backbone traffic monitoring with anomaly detection

ADAM & RAL: Adaptive Memory Learning and Reinforcement Active Learning for Network Monitoring

Comparative Study between Big Data Analysis Techniques in Intrusion Detection

Contact Info

Product

Resources

About