Bypassing Space Explosion in High-Speed Regular Expression Matching

Abstract

Abstract-Network intrusion detection and prevention systems commonly use regular expression (RE) signatures to represent individual security threats. While the corresponding DFA for any one RE is typically small, the DFA that corresponds to the entire set of REs is usually too large to be constructed or deployed. To address this issue, a variety of alternative automata implementations that compress the size of the final automaton have been proposed such as XFA and D 2 FA. The resulting final automata are typically much smaller than the corresponding DFA. However, the previously proposed automata construction algorithms do suffer from some drawbacks. First, most employ a "Union then Minimize" framework where the automata for each RE are first joined before minimization occurs. This leads to an expensive NFA to DFA subset construction on a relatively large NFA. Second, most construct the corresponding large DFA as an intermediate step. In some cases, this DFA is so large that the final automaton cannot be constructed even though the final automaton is small enough to be deployed. In this paper, we propose a "Minimize then Union" framework for constructing compact alternative automata focusing on the D 2 FA. We show that we can construct an almost optimal final D 2 FA with small intermediate parsers. The key to our approach is a space and time efficient routine for merging two compact D 2 FA into a compact D 2 FA. In our experiments, our algorithm runs on average 155 times faster and uses 1500 times less memory than previous algorithms. For example, we are able to construct a D 2 FA with over 80,000,000 states using only 1GB of main memory in only 77 minutes.

RICS‐DFA: a space and time‐efficient signature matching algorithm with Reduced Input Character Set

“…Patel et al . has found another drawback of automata construction algorithm where multiple NAFs are joined into a large DFA and then minimize it. That leads to an expensive subset construction from NFAs to DFAs for large NFAs and maybe a small final minimized DFA cannot be generated because of the failure of subset construction.…”

“…Compared with D 2 FA, A-DFA results in at most 2N.k C 1/=k state traversals when processing an input string of length N, k being a positive integer, and yields a 10fold improvement in compression ratio. Patel et al [10] has found another drawback of automata construction algorithm where multiple NAFs are joined into a large DFA and then minimize it. That leads to an expensive subset construction from NFAs to DFAs for large NFAs and maybe a small final minimized DFA cannot be generated because of the failure of subset construction.…”

O3fa

“…Existing proposals targeting DFA-based solutions have focused on two aspects: (i) designing compression schemes aimed at minimizing the DFA memory footprint; and (ii) devising new automata to alternate DFAs in case of state explosion. Alphabet reduction [5,[22][23][24], run-length encoding [5], default transition compression [22,25], state merging [7] and delta-FAs [26] fall in the first category, while multiple-DFAs [5,6], hybrid-FAs [7], history-based-FAs [8], XFAs [9], counting-FAs [10], and JFAs [11] fall in the second category. All of these solutions, however, have been designed to operate on reassembled packet streams.…”

Offset-FA: A Uniform Method to Handle Both Unbounded and Bounded Repetitions in Regular Expression Matching

“…Most studies have focused on compression technologies [ 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 ] to restrain the DFA expansion problem. Classical compression algorithms such as D FA [ 25 ] and A-DFA [ 26 ] were devised to reduce the space cost of DFA with performance guarantees.…”