Detecting anomalous packets in network transfers: investigations using PCA, autoencoder and isolation forest in TCP

Abstract

Large-scale scientific workflows rely heavily on high-performance file transfers. These transfers require strict quality parameters such as guaranteed bandwidth, no packet loss or data duplication. To have successful file transfers, methods such as predetermined thresholds and statistical analysis need to be done to determine abnormal patterns. Network administrators routinely monitor and analyze network data for diagnosing and alleviating these, making decisions based on their experience. However, as networks grow and become complex, monitoring large data files and quickly processing them, makes it improbable to identify errors and rectify these. Abnormal file transfers have been classified by simply setting alert thresholds, via tools such as PerfSonar and TCP statistics (Tstat). This paper investigates the feasibility of unsupervised feature extraction methods for identifying network anomaly patterns with three unsupervised classification methods-principal component analysis, autoencoder and isolation forest. We collect file transfer statistics from two experiment sets-synthetic iPerf generated traffic and 1000 Genome workflow runs, with synthetically introduced anomalies. Our results show that while PCA and a simple autoencoder finds it difficult to detect clusters, the tree-variant isolation forest is able to identify anomalous packets by breaking down TCP traces into tree classes early.

Minimum cost‐compression risk in principal component analysis

“…2009), network anomaly (Kiran et al . 2020), and others. PCA algorithm has been modified to deal with robustness issue as well (refer to Tarr, Müller & Weber 2016 and related references).…”

“…These PCs are then arranged in order of decreasing importance based on their variability and only those which contribute significantly to the representation of the dataset are retained. The PCA technique has found applications in areas such as script recognition (Naser et al 2010), missing data (Cardot & Degras 2018), face recognition (Dryden et al 2009), network anomaly (Kiran et al 2020), and others. PCA algorithm has been modified to deal with robustness issue as well (refer to Tarr, Muller & Weber 2016 and related references).…”

Unsupervised Abnormal Traffic Detection through Topological Flow Analysis

“…Several wide-range anomaly detection techniques that are often used in network IDSs are listed as follows: Statistical Profiling with Histograms [3], [4], Parametric and Non-parametric Statistical Modeling [5], [6], (Deep or Shallow) Artificial Neural Networks and Autoencoders [7]- [10], (One-Class) Support Vector Machines [11], [12], Reconstruction methods [13]- [15], Clustering methods [16], [17]. However, generally most of these learning systems detect abnormal data flows or packets based on their features and characteristics.…”

Mining Workflows for Anomalous Data Transfers

“…We choose to use CART for its explainability and efficiency, as discussed in [20,59,34]. RF was used in the previous study for this same problem for its advantages of performance [60]. However, we will show later in this paper that RF without optimization is not enough.…”