Vertical, Temporal, and Horizontal Scaling of Hierarchical Hypersparse GraphBLAS Matrices

Abstract

Defending community-owned cyber space requires community-based efforts. Large-scale network observations that uphold the highest regard for privacy are key to protecting our shared cyberspace. Deployment of the necessary network sensors requires careful sensor placement, focusing, and calibration with significant volumes of network observations. This paper demonstrates novel focusing and calibration procedures on a multi-billion packet dataset using high-performance GraphBLAS anonymized hypersparse matrices. The run-time performance on a real-world data set confirms previously observed realtime processing rates for high-bandwidth links while achieving significant data compression. The output of the analysis demonstrates the effectiveness of these procedures at focusing the traffic matrix and revealing the underlying stable heavy-tail statistical distributions that are necessary for anomaly detection. A simple model of the corresponding probability of detection (p d ) and probability of false alarm (p fa ) for these distributions highlights the criticality of network sensor focusing and calibration. Once a sensor is properly focused and calibrated it is then in a position to carry out two of the central tenets of good cybersecurity:(1) continuous observation of the network and (2) minimizing unbrokered network connections.

Temporal Correlation of Internet Observatories and Outposts

“…Analysis of such a large network data set is computationally challenging [25]- [27]. Using the combined resources of the Supercomputing Centers at UC San Diego, Lawrence Berkeley National Laboratory, and MIT, the spatial temporal structure of anonymized source-destination pairs from the CAIDA Telescope data has been analyzed leveraging prior work on massively parallel GraphBLAS and D4M hierarchical hypersparse matrices [28]- [35] to reveal a wide range of scaling relations [36]. For this study 5 contiguous subsets of 2 30 CAIDA Telescope packets were selected and formed into GraphBLAS hypersparse traffic matrices at approximately 6-week intervals (see Table I).…”