Stream-Oriented Network Traffic Capture and Analysis for High-Speed Networks

Abstract

Abstract-Intrusion detection, traffic classification, and other network monitoring applications need to analyze the captured traffic beyond the network layer to allow for connection-oriented analysis, and achieve resilience to evasion attempts based on TCP segmentation. Existing network traffic capture frameworks, however, provide applications with raw packets and leave complex operations like flow tracking and TCP stream reassembly to application developers. This gap, between what applications need and what systems provide, leads to increased application complexity, longer development time, and most importantly, reduced performance due to excessive data copies between the packet capture subsystem and the stream processing module.This paper presents the Stream capture library (Scap), a network monitoring framework built from the ground up for stream-oriented traffic processing. Based on a kernel module that directly handles flow tracking and TCP stream reassembly, Scap delivers to user-level applications flow-level statistics and reassembled streams by minimizing data movement operations and discarding uninteresting traffic at early stages, while it inherently supports parallel processing on multi-core architectures, and uses advanced capabilities of modern network cards. Our experimental evaluation shows that Scap can capture all streams for traffic rates two times higher than other stream reassembly libraries. Finally, we present the implementation and performance evaluation of four popular network traffic monitoring applications built on top of Scap.

A Survey on Network Security-Related Data Collection Technologies

“…For example, Woods proposed to use a shared memory to exchange network data packets between kernel space and user space in order to avoid a large number of replication and interrupt operations, thus improving packet capture efficiency [66]. Papsadogiannakis et al [30] introduced a technique called subzero packet copy that can avoid copying uninteresting packets across different memory areas and another technique named prioritized packet loss that can be adapted to overload conditions by dropping the packets with lower priority. Moreover, they proposed a streamoriented network monitoring library named Stream capture library (Scap) based on the proposed two technologies.…”

“…The use of the underlying NIC is inevitable for any kinds of traffic capture technologies. Network cards and corresponding software applications can be customized in order to achieve special purposes [2], [26], [29], [30]. Unfortunately, the capturing tools relying on standard NICs have several limitations.…”

“…Their simulation results showed that the proposed scheme significantly reduces the ratio of packet loss and improves the efficiency of data collection system. Moreover, Scap in [30] used a dynamic load balancing mechanism, which applies flow director filters to balance the network traffic load across available queues and cores. It also applies a technique named Receive Side Scaling (RSS) that uses a hash function based on a 5-tuple of packets.…”

A survey of bitmap index compression algorithms for Big Data

“…Even after polymerization, there are still millions of flow records. Though only dealing with flow data, a central point of an operator would generate millions of records per second which will reach 30 billion per day [10][11][12][13] . Therefore, how to capture and store high arriving speed of packets and streams in real time is a major challenge.…”

High-speed TCP flow record extraction using GPUs

“…This is a usual technique which has also been used in other works [29]. Note that if two different connections, possibly from a reincarnation, have the same source and destination IP address and TCP ports, then it is not possible to associate each segment to each particular connection.…”