Which Regular Languages can be Efficiently Indexed?

Abstract

Consider the problem of matching a pattern P of length π against the elements of a given regular language L in the setting where L can be pre-processed off-line in a fast data structure (an index). Regular expression matching is an ubiquitous problem in computer science, finding fundamental applications in areas including, but not limited to, natural language processing, search engines, compilers, and databases. Recent results have settled the exact complexity of this problem: Θ(πm) time is necessary and sufficient for indexed pattern matching queries, where m is the size of an NFA recognizing L. This, however, does not mean that all regular languages are hard to index: for instance, for the sub-class of Wheeler languages [SODA'20] we can reduce query time to the optimal O(π). A Wheeler language admits a total order of a finite refinement of its Myhill-Nerode equivalence classes reflecting the co-lexicographic order of their elements. This boosts indexing performance because classes whose elements are suffixed by P form a range in this order. In [SODA'21], this technique was extended to arbitrary NFAs by allowing the order to be partial. This line of attack suggested that the width p of such an order is the parameter ultimately capturing the fine-grained complexity of the problem: (i) indexed pattern matching can always be solved in O(πp 2 ) time, (ii) the standard powerset construction algorithm always produces an output whose size is exponential in p rather than in the input's size, and (iii) p even determines how succinctly NFAs can be encoded.In the present work, we take a step further and study the hierarchy of p-sortable languages: regular languages accepted by automata of width p. Our main contributions are the following: (i) we show that the hierarchy is strict and does not collapse, (ii) we provide (exponential) upper and lower bounds relating the minimum widths of equivalent NFAs and DFAs, and (iii) we characterize DFAs of minimum p for a given L via a co-lexicographic variant of the Myhill-Nerode theorem. Our findings imply that in polynomial time we can build an index breaking the worst-case conditional lower bound of Ω(πm), whenever the input NFA's width is at most ϵ log 2 m, for any constant 0 ≤ ϵ < 1/2.