Semantics and algorithms for data-dependent grammars

Abstract

Traditional parser generation technologies are incapable of handling the demands of modern programmers. In this paper, we present the design and theory of a new parsing engine, YAKKER, capable of handling the requirements of modern applications including (1) full scannerless context-free grammars with (2) regular expressions as right-hand sides for defining nonterminals. YAKKER also includes (3) facilities for binding variables to intermediate parse results and (4) using such bindings within arbitrary constraints to control parsing. These facilities allow the kind of data-dependent parsing commonly needed in systems applications, particularly those that operate over binary data. In addition, (5) nonterminals may be parameterized by arbitrary values, which gives the system good modularity and abstraction properties in the presence of data-dependent parsing. Finally, (6) legacy parsing libraries, such as sophisticated libraries for dates and times, may be directly incorporated into parser specifications. We illustrate the importance and utility of this rich format specification language by presenting its use on examples ranging from difficult programming language grammars to web server logs to binary data specification. We also show that our grammars have important compositionality properties and explain why such properties are important in modern applications such as automatic grammar induction.In terms of technical contributions, we provide a traditional high-level semantics for our new grammar formalization and show how to compile these grammars into nondeterministic automata. These automata are stack-based, somewhat like conventional pushdown automata, but are also equipped with environments to track data-dependent parsing state. We prove the correctness of our translation of data-dependent grammars into these new automata and then show how to implement the automata efficiently using a variation of Earley's parsing algorithm.

Parsing with Earley Virtual Machines

“…By applying a variation subset construction to the generated Earley graphs, the authors of [2] achieve additional performance gains. Finally, a futher variation of Efficient Earley parser called Yakker [3] enables the use of data-dependant constraints within the grammars. An Earley parser variation described in [4] moves in a different direction: the authors of [4] propose a version of Earley parser that is suitable for parsing reflective (more often referred as adaptable) grammars, that can be modified during parsing to augment the input language.…”

“…Therefore, EVM 3 can be used to recognize data-dependant language constructs that cannot be parsed by more traditional parsing methods. In section III we present two separate methods for constructing the abstract syntax trees (or more precisely, shared packed parse forests) based on the input data.…”

A New Method for Dependent Parsing

“…In earlier work, we presented a formalism that incorporated support for dependent parsing, scannerless parsing, full context-free grammars and foreign parsers [10]. That work focused on the correctness of the translation from grammars to transducers and their execution using an Earley-style algorithm.…”

“…We have compared many of the closely related formalisms with our dependent grammars in earlier work [10]. However, we add here a brief comparison with definite clause grammars (DCGs) [15], a popular formalism for specifying grammars as logic programs.…”

“…In previous work, we used dependent grammars to cleanly specify contextsensitive syntax [10]. Dependent grammars extend context-free grammars by allowing semantic values to be bound to variables and used to guide subsequent parsing.…”