Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study

Abstract

Numerous research studies have demonstrated the effectiveness of machine learning techniques in application to network intrusion detection. And yet, the adoption of machine learning for securing large-scale network environments remains limited. The community acknowledges that network security presents unique challenges for machine learning, and the lack of training data representative of modern traffic remains one of the most intractable issues. New attempts are continuously made to develop high quality benchmark datasets and proper data collection methodologies. The CICIDS2017 dataset is one of the recent results, created to meet the demanding criterion of representativeness for network intrusion detection. In this paper, we revisit CICIDS2017 and its data collection pipeline and analyze correctness, relevance and overall utility of the dataset for the learning task. In the process, we uncover a series of traffic generation, flow extraction and labeling problems that severely affect the validity of data. To counter these issues, we investigate and eliminate their causes and regenerate the dataset. As a result, more than 20% of traffic traces are reconstructed or relabeled, and machine learning benchmarks demonstrate significant improvements. Our study exemplifies how data collection issues may have enormous impact on model evaluation and provides recommendations for their anticipation and prevention.

SoK: The Impact of Unlabelled Data in Cyberthreat Detection

“…We assume that all the considered datasets are correctly labelled. However, such assumption may be overly optimistic: as described in §2, manual labelling is an error-prone task, and some recent papers highlighted that even well-known datasets may contain flaws (e.g., [151]). Due to the seminal nature of this SoK paper, we do not make any change to the provided labels-which also facilitates comparisons with previous works using such datasets, as the ground truth is the same.…”

The Role of Machine Learning in Cybersecurity

“…Despite the recent Grammatopoulos, and Fabio Di Franco interest in ML led to the release of more open datasets (e.g., [10,31,142]), such datasets exhibit limitations [163]. For instance, inaccurate labels, fast obsolescence, small and synthetic environments (e.g., [115]), or even flawed generation process-as shown in [66]. All these problems can only be mitigated to some degree (e.g.…”

Gotham Testbed: a Reproducible IoT Testbed for Security Experiments and Dataset Generation

“…Many of the defined requirements for testbeds also overlap with the criteria that network security datasets should meet [13,17,18]. The criteria can be summarized as follows: the dataset provides real and complete network traces; the traffic is generated using a valid network topology that includes clients, servers and network equipment; the dataset is labeled to distinguish between benign and malicious traces; highly heterogeneous regarding included services, network protocols, normal and attack behaviors; easily extendable; reproducible; shareable and documented.…”

“…Besides offering different types of attacks, diversity within the same attack type should also be provided. This procedure includes combining different attack tools that perform similar actions and using multiple options and flags for each attack [17].…”

“…The scarcity of details on the devices, software versions, and configuration files or command-line arguments used in attacking tools, hinders the reproducibility of the datasets. This issue was recently raised in [17] when analyzing the popular CICIDS2017 [18] dataset.…”