Cyber anomaly detection: Using tabulated vectors and embedded analytics for efficient data mining

Abstract

Firewalls, especially at large organizations, process high velocity internet traffic and flag suspicious events and activities. Flagged events can be benign, such as misconfigured routers, or malignant, such as a hacker trying to gain access to a specific computer. Confounding this is that flagged events are not always obvious in their danger and the high velocity nature of the problem. Current work in firewall log analysis is manual intensive and involves manpower hours to find events to investigate. This is predominantly achieved by manually sorting firewall and intrusion detection/prevention system log data. This work aims to improve the ability of analysts to find events for cyber forensics analysis. A tabulated vector approach is proposed to create meaningful state vectors from time-oriented blocks. Multivariate and graphical analysis is then used to analyze state vectors in human-machine collaborative interface. Statistical tools, such as the Mahalanobis distance, factor analysis, and histogram matrices, are employed for outlier detection. This research also introduces the breakdown distance heuristic as a decomposition of the Mahalanobis distance, by indicating which variables contributed most to its value. This work further explores the application of the tabulated vector approach methodology on collected firewall logs. Lastly, the analytic methodologies employed are integrated into embedded analytic tools so that cyber analysts on the front-line can efficiently deploy the anomaly detection capabilities.

Security Methods for Critical Infrastructure Communications

“…Due to the expanding variety and volume of devices in IoT CI implementations, future CI networks themselves have characteristics seen in the 3 V's (volume, variety, and velocity) of big data (Bihl, Young, & Weckman, 2016). Thus, monitoring logs and transmissions of communication devices to find threats can involve big data analytics due the massive amount of events logged (Samuelson, 2016) (Gutierrez, Bauer, Boehmke, Saie, & Bihl, 2017).…”

“…Incorporating intrusion detection and prevention systems (IDPS) into industrial control networks can mitigate MAC related attacks and provide a log of events which violate access rules (Zhu & Sastry, 2010) (Xing, Srinivasan, Jose, Li, & Cheng, 2010). However, IDPS systems generally rely on coded rules, which are limited against new and novel attacks (Gutierrez, Bauer, Boehmke, Saie, & Bihl, 2017). A variety of network based routing attacks exist and these can take the form of attackers flooding, or corrupting routing information or flooding the network with replicated packets to consume bandwidth and cause communication termination (Xing, Srinivasan, Jose, Li, & Cheng, 2010).…”

Topological Data Analysis for Enhancing Embedded Analytics for Enterprise Cyber Log Analysis and Forensics

“…Additionally, due to their scale and the sensitivity of the data and operations they support, enterprise cyber security involves many layers and efforts, such as encompasses multiple, very large local networks, which have their own devices, standards, and administration. Enterprise cyber security often includes forensics analysis and security or data fusion centers to detect emerging threats [3] [4].…”

“…Since enterprise level cyber systems must provide accountability and look for emerging threats, cyber forensics of firewall and IDPS logs is a useful component of any enterprise-level cyber security operations [10]. However, the unstructuredness of cyber logs in enterprises results in significant manual analysis for cyber forensics [3] [10].…”

“…Herein, statistical and visual methods are combined into an embedded application. For this, the authors leverage their past work [3] in cyber analytics and employ a tabulated feature vector (TFV) approach to process log files and identify anomalies. In the TFV approach, log files are divided into temporal blocks and analysts are cued to blocks of interest.…”