Fast, Flexible, and Declarative Construction of Abstract Syntax Trees with PEGs

Proceedings of the 2016 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software

2016

Self Cite

Nez is a PEG(Parsing Expressing Grammar)-based open grammar language that allows us to describe complex syntax constructs without action code. Since open grammars are declarative and free from a host programming language of parsers, software engineering tools and other parser applications can reuse once-defined grammars across programming languages.A key challenge to achieve practical open grammars is the expressiveness of syntax constructs and the resulting parser performance, as the traditional action code approach has provided very pragmatic solutions to these two issues. In Nez, we extend the symbol-based state management to recognize context-sensitive language syntax, which often appears in major programming languages. In addition, the Abstract Syntax Tree constructor allows us to make flexible tree structures, including the left-associative pair of trees. Due to these extensions, we have demonstrated that Nez can parse not all but many grammars.Nez can generate various types of parsers since all Nez operations are independent of a specific parser language. To highlight this feature, we have implemented Nez with dynamic parsing, which allows users to integrate a Nez parser as a parser library that loads a grammar at runtime. To achieve its practical performance, Nez operators are assembled into low-level virtual machine instructions, including automated state modifications when backtracking, transactional controls of AST construction, and efficient memoization in packrat parsing. We demonstrate that Nez dynamic parsers achieve very competitive performance compared to existing efficient parser generators.

“…Note that efficient packrat parsing with ASTs requires additional transactional management for ASTs. A detailed mechanism is reported in [22].…”

Section: Parser Runtime and Parser Generationmentioning

confidence: 99%

Nez: practical open grammar language

Proceedings of the 2016 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software

2016

Self Cite

“…matching capability with symbol table handlers and parsing conditions, due to the space constraints. Further information on the AST construction can be referred to our report [11]. Symbol table is a global state used to maintain strings, whose meaning is specialized in parsing contexts.…”

Section: Overviewmentioning

confidence: 99%

A Declarative Extension of Parsing Expression Grammars for Recognizing Most Programming Languages

Matsumura

Journal of Information Processing

2016

Self Cite

Parsing Expression Grammars are a popular foundation for describing syntax. Unfortunately, several syntax of programming languages are still hard to recognize with pure PEGs. Notorious cases appears: typedef-defined names in C/C++, indentation-based code layout in Python, and HERE document in many scripting languages. To recognize such PEG-hard syntax, we have addressed a declarative extension to PEGs. The "declarative" extension means no programmed semantic actions, which are traditionally used to realize the extended parsing behavior. Nez is our extended PEG language, including symbol tables and conditional parsing. This paper demonstrates that the use of Nez Extensions can realize many practical programming languages, such as C, C#, Ruby, and Python, which involve PEG-hard syntax.

“…Many implemented parser generators Ford (2014) (2008); Kuramitsu (2016) that can load PEGs at runtime to parse. Since the dynamic parsing can pass through the code generation and its compilation process, we can use it as an existing XML validator.…”

Section: Parsing Algorithmmentioning

confidence: 99%

XML schema validation using parsing expression grammars

Hamaguchi

2015

Preprint

Self Cite

Schema validation is an integral part of reliable information exchange on the Web. However, implementing an efficient schema validation tool is not easy. We highlight the use of parsing expression grammars (PEGs), a recognition-based foundation for describing syntax, and apply it to the XML/DTD validation. This paper shows that structural schema constraints in document type definitions (DTDs) can be validated by the converted PEGs with the linear time and constant space consumption. We study the performance of several existing PEG-based tools, and then confirm that the converted PEGs achieve a practical and even competitive level of performance under existing standard XML/DTD validators.