Memory-Efficient Regular Expression Search Using State Merging

Abstract

Abstract-Pattern matching is a crucial task in several critical network services such as intrusion detection and policy management. As the complexity of rule-sets increases, traditional string matching engines are being replaced by more sophisticated regular expression engines. To keep up with line rates, deal with denial of service attacks and provide predictable resource provisioning, the design of such engines must allow examining payload traffic at several gigabits per second and provide worst case speed guarantees. While regular expression matching using deterministic finite automata (DFA) is a well studied problem in theory, its implementation either in software or specialized hardware is complicated by prohibitive memory requirements. This is especially true for DFAs representing complex regular expressions present in practical rule-sets.In this paper, we introduce a novel method to drastically reduce the DFA memory requirement and still provide worst-case speed guarantees. Specifically, we merge several "non-equivalent" states in a DFA by introducing labels on their input and output transitions. We then propose a data structure to represent the merged states and the transition labels. We show that, with very few assumptions about the original DFA, such a transformation results in significant compression in the DFA representation. We have implemented a state merging and transition labeling algorithm for DFAs, and show that for Snort and Bro security rule-sets, state merging results in memory reductions of an order of magnitude.

Fast and Memory-Efficient Regular Expression Matching Using Transition Sharing

“…But the basic bitmap-based data structures do not take advantage of duplicate entries in the transition table. Michela Becchi et al extend the bitmap structure [10] . The content label's fields consists of a state discriminator, the list of characters for which a labeled transition is defined and an identifier for the default transition state.…”

“…Michela Becchi et al [10] propose state merging method that can both reduce the number of states and transitions. This method merges several even "non-equivalent" states by introducing labels.…”

“…2. The example and data structure are all same as that of in [10]. In order to avoid cluster, all our figures exclude the transitions leading to the starting state (state 0) and the bitmap of each state.…”

“…In order to better illustrate the memory reduction of our approach and make comparison with other approaches, for each group regular expressions, we compiled them into a single DFA instead of using the set splitting techniques proposed in [10]. The space and time requirements of each approach are showed in the Table 3.…”

A Regular Expression Matching Algorithm Based on High-Efficient Finite Automaton

“…In accelerating the matching speed of a regular expres-sion aspect, Kumar et al [3] proposed a D2FA regular expression matching algorithm in order to accelerate the matching speed. Becchi and Cadambi [4] suggested a DFA regular expression matching algorithm based on state fusion, making use of a migration edge marking method in order to integrate a multiple non-equivalent state for guaranteeing DFA performance. In Brodie method, Brodie et al [5] presented a new way that can increase the throughput of regular expression matching by expanding the alphabet set; however, this method could result in an exponential growth of memory requirement in the worst case.…”