Challenges and Pitfalls in Generating Representative ICS Datasets in Cyber Security Research

Mitseva, Asya; Thierse, Paul; Hoffmann, H.; Er, Devran; Panchenko, Andriy

doi:10.1007/978-3-031-25460-4_22

Cited by 1 publication

(1 citation statement)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lastly, reviews of datasets' quality [49], e.g., with statistical means accessing the data distribution and stability [79] or analysis of how easily a dataset can be "solved" [86], can guide developers in choosing a relevant dataset. For more concrete advice on how scientific IIDS evaluation datasets should be designed, please refer to the works by Gómez et al [64] and Mitseva et al [60].…”

Section: Recommendations For Iids Evaluationsmentioning

confidence: 99%

[SoK] Evaluations in Industrial Intrusion Detection Research

Lamberts,

Wolsing,

Wagner

et al. 2023

Journal of Systems Research

View full text Add to dashboard Cite

Industrial systems are increasingly threatened by cyberattacks with potentially disastrous consequences. To counter such attacks, industrial intrusion detection systems strive to timely uncover even the most sophisticated breaches. Due to its criticality for society, this fast-growing field attracts researchers from diverse backgrounds, resulting in 130 new detection approaches in 2021 alone. This huge momentum facilitates the exploration of diverse promising paths but likewise risks fragmenting the research landscape and burying promising progress. Consequently, it needs sound and comprehensible evaluations to mitigate this risk and catalyze efforts into sustainable scientific progress with real-world applicability. In this paper, we therefore systematically analyze the evaluation methodologies of this field to understand the current state of industrial intrusion detection research. Our analysis of 609 publications shows that the rapid growth of this research field has positive and negative consequences. While we observe an increased use of public datasets, publications still only evaluate 1.3 datasets on average, and frequently used benchmarking metrics are ambiguous. At the same time, the adoption of newly developed benchmarking metrics sees little advancement. Finally, our systematic analysis enables us to provide actionable recommendations for all actors involved and thus bring the entire research field forward.

show abstract