A fast and accurate threat detection and prevention architecture using stream processing

Abstract

Late detection of security breaches increases the risk of irreparable damages and limits any mitigation attempts. We propose a fast and accurate threat detection and prevention architecture that combines the advantages of real-time streaming with batch processing over a historical database. We create a dataset by capturing both legitimate and malicious traffic and propose two ways of combining packets into flows, one considering a time window and the other analyzing the first few packets of each flow per period. We also investigate the effectiveness of our proposal on real-world network traces obtained from a significant Brazilian network operator providing broadband Internet to their customers. We implement and evaluate three classification algorithms and two anomaly detection methods. The results show an accuracy higher than 95% and an excellent trade-off between attack detection and false-positive rates. We further propose an improved scheme based on software defined networks that automatically prevents threats by analyzing only the first few packets of a flow. The proposal promptly and efficiently blocks threats, is robust, and can scale up, even when the attacker employs spoofed IP.

Lc‐Stream: An elastic scheduling strategy with latency constraints in geo‐distributed stream computing environments

“…Common big data real‐time stream computing systems such as Storm, 4 Spark Streaming 5 and Flink 6 play an important role in many fields, including online system monitoring, 7 mobile sensing data processing 8 and the Internet of Things, 9 financial risk control, 10 recommendation systems, 11 and threat detection 12 …”

“…Common big data real-time stream computing systems such as Storm, 4 Spark Streaming 5 and Flink 6 play an important role in many fields, including online system monitoring, 7 mobile sensing data processing 8 and the Internet of Things, 9 financial risk control, 10 recommendation systems, 11 and threat detection. 12 Storm is a relatively low latency real-time distributed stream processing framework that comes with a default scheduling scheme. Storm uses a pooling method in its allocation process, which often causes load skew.…”