An unsupervised approach for traffic trace sanitization based on the entropy spaces

Alvarado, Pablo; Vargas-Rosales, César; Martínez-Peláez, Rafael; Toral-Cruz, Homero; Martínez-Herrera, Alberto F.

doi:10.1007/s11235-015-0017-6

Cited by 10 publications

(6 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Real traffic, on the other hand, should be stripped of confidential data, carefully labeled and rigorously sanitized in order to meet the established criteria. Several studies contributed approaches to sanitization of traffic [13,25,47,100] in order to not only label embedded attacks and benign traces, but also to pre-select the most representative instances. Automated sanitization uses such methods as entropy analysis and signature-based attack labeling, which may result in erroneous ground-truth.…”

Section: Representative Datasets and Ground Truthmentioning

confidence: 99%

Open-World Network Intrusion Detection

Rimmer

Nadeem

Verwer

et al. 2022

Security and Artificial Intelligence

View full text Add to dashboard Cite

This chapter contributes to the ongoing discussion of strengthening security by applying AI techniques in the scope of intrusion detection. The focus is set on open-world detection of attacks through data-driven network traffic analysis. This research topic is complementary to the earlier chapter on intelligent malware detection.In this chapter, we revisit the foundations of machine learning-based solutions for network security, which aim to make network defense tools more autonomous, adaptive, proactive and responsive. Specifically, we give a comprehensive introduction to the research on anomaly detection for network intrusion detection -that is, defensive schemes that do not assume complete prior knowledge of malicious patterns and instead learn the notion of normality from benign traffic. Along with outlining the recent advances in the field, we provide insights and reflect on the current limitations and research challenges. Therefore, this chapter presents compelling research opportunities to advance machine learning techniques in network security and push the boundaries of open-world network intrusion detection.

show abstract

Section: Representative Datasets and Ground Truthmentioning

confidence: 99%

Open-World Network Intrusion Detection

Rimmer

Nadeem

Verwer

et al. 2022

Security and Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…In [23], the authors propose techniques for instrumenting network warfare competitions to collect scientifically valid labeled datasets, which otherwise would be resource-intensive. Similarly, Velarde-Alvarado et al [11] remark the scarcity of suitable datasets for AIDS development and propose a semi-automated process for the sanitization of the traffic captured based on the entropy of embedded traffic flows. This enables the collection of large volumes of data without excessive resource consumption in terms of manual supervision or computational resources.…”

Section: Related Workmentioning

confidence: 99%

“…However, the workload associated with this process grows linearly with the size of the trace. And, although unsupervised sanitization approaches have been suggested (e.g., analysis of entropy [11], or filtering known-attacks with signature-based IDS [12]), manual supervision may be unavoidable in order to discover attacks (e.g., 0-day) unnoticed by fully automated methods [13], [14].…”

Section: Introductionmentioning

confidence: 99%

How Much Training Data is Enough? A Case Study for HTTP Anomaly-Based Intrusion Detection

et al. 2020

View full text Add to dashboard Cite

Most anomaly-based intrusion detectors rely on models that learn from training datasets whose quality is crucial in their performance. Albeit the properties of suitable datasets have been formulated, the influence of the dataset size on the performance of the anomaly-based detector has received scarce attention so far. In this work, we investigate the optimal size of a training dataset. This size should be large enough so that training data is representative of normal behavior, but after that point, collecting more data may result in unnecessary waste of time and computational resources, not to mention an increased risk of overtraining. In this spirit, we provide a method to find out when the amount of data collected at the production environment is representative of normal behavior in the context of a detector of HTTP URI attacks based on 1-grammar. Our approach is founded on a set of indicators related to the statistical properties of the data. These indicators are periodically calculated during data collection, producing time series that stabilize when more training data is not expected to translate to better system performance, which indicates that data collection can be stopped. We present a case study with real-life datasets collected at the University of Seville (Spain) and a public dataset from the University of Saskatchewan. The application of our method to these datasets showed that more than 42% of one trace, and almost 20% of another were unnecessarily collected, thereby showing that our proposed method can be an efficient approach for collecting training data at the production environment. INDEX TERMS Anomaly-based intrusion detection, dataset assessment, training.

show abstract

“…In order to characterise and detect DDoS attacks, they propose the use of machine learning techniques along with entropy-based features, since these features measure information uncertainties and thus can show changes in traffic patterns. This work is related to [13] where the authors claim that the accuracy and reliability of anomaly-based network intrusion detection systems depend on the quality of the data used to build normal behaviour profiles; thus, the authors use entropy measurements and machine learning techniques such as clustering and classification in order sanitise network logs to improve the quality of these profiles.…”

Section: Related Workmentioning

confidence: 99%

DNS-ADVP: A Machine Learning Anomaly Detection and Visual Platform to Protect Top-Level Domain Name Servers Against DDoS Attacks

et al. 2019

View full text Add to dashboard Cite

DNS DDoS attacks may severely affect the operation of computer networks, prompting the need for methods able to timely detect them, and then to apply mitigation countermeasures. Visual models have been used to detect an ongoing DDoS attack, but often demand continuous attention from IT staff. However, machine learning techniques could complement a visual model with further information and with on-time alerts that could help IT officers give attention only when an attack is in progress at its very early stage. In this paper, we present DNS-ADVP, a DNS Anomaly Detection Visual Platform, which, in an integrated manner, provides a novel visualisation that depicts on-line DNS traffic, and a one-class classifier that deals with traffic anomaly detection. Using the visual mode, an IT officer may interpret the current state of traffic for an authoritative DNS server; the model comes with visual semaphores, controlled by the one-class classifier. Due to the highly dynamic nature of DNS traffic, our classification method continuously updates what counts as normal behaviour; it has been successfully tested on synthetic attacks, with an 83% of the area under the curve (AUC). DNS-ADVP is currently in use to real-time monitoring an actual authoritative DNS server.

show abstract

An unsupervised approach for traffic trace sanitization based on the entropy spaces

Cited by 10 publications

References 30 publications

Open-World Network Intrusion Detection

Open-World Network Intrusion Detection

How Much Training Data is Enough? A Case Study for HTTP Anomaly-Based Intrusion Detection

DNS-ADVP: A Machine Learning Anomaly Detection and Visual Platform to Protect Top-Level Domain Name Servers Against DDoS Attacks

Contact Info

Product

Resources

About