Concrete Syntax with Black Box Parsers

Abstract

Context: Meta programming consists for a large part of matching, analyzing, and transforming syntax trees. Many meta programming systems process abstract syntax trees, but this requires intimate knowledge of the structure of the data type describing the abstract syntax. As a result, meta programming is errorprone, and meta programs are not resilient to evolution of the structure of such ASTs, requiring invasive, fault-prone change to these programs.Inquiry: Concrete syntax patterns alleviate this problem by allowing the meta programmer to match and create syntax trees using the actual syntax of the object language. Systems supporting concrete syntax patterns, however, require a concrete grammar of the object language in their own formalism. Creating such grammars is a costly and error-prone process, especially for realistic languages such as Java and C++.Approach: In this paper we present Concretely, a technique to extend meta programming systems with pluggable concrete syntax patterns, based on external, black box parsers. We illustrate Concretely in the context of Rascal, an open-source meta programming system and language workbench, and show how to reuse existing parsers for Java, JavaScript, and C++. Furthermore, we propose Tympanic, a DSL to declaratively map external AST structures to Rascal's internal data structures. Tympanic allows implementors of Concretely to solve the impedance mismatch between object-oriented class hierarchies in Java and Rascal's algebraic data types. Both the algebraic data type and AST marshalling code is automatically generated.Knowledge: The conceptual architecture of Concretely and Tympanic supports the reuse of pre-existing, external parsers, and their AST representation in meta programming systems that feature concrete syntax patterns for matching and constructing syntax trees. As such this opens up concrete syntax pattern matching for a host of realistic languages for which writing a grammar from scratch is time consuming and error-prone, but for which industry-strength parsers exist in the wild.Grounding: We evaluate Concretely in terms of source lines of code (SLOC), relative to the size of the AST data type and marshalling code. We show that for real programming languages such as C++ and Java, adding support for concrete syntax patterns takes an effort only in the order of dozens of SLOC. Similarly, we evaluate Tympanic in terms of SLOC, showing an order of magnitude of reduction in SLOC compared to manual implementation of the AST data types and marshalling code.Importance: Meta programming has applications in reverse engineering, reengineering, source code analysis, static analysis, software renovation, domain-specific language engineering, and many others. Processing of syntax trees is central to all of these tasks. Concrete syntax patterns improve the practice of constructing meta programs. The combination of Concretely and Tympanic has the potential to make concrete syntax patterns available with very little effort, thereby improving and promoting the application of ...

High-fidelity metaprogramming with separator syntax trees

“…This opens the way to high-fidelity metaprogramming in systems like Rascal or TXL, when parsing is realized by reusing external, black box parsers (cf. [1]).…”

“…The following invariants on SST nodes hold: If we now have a source text (1), which corresponds to the AST paren(lit("1")) over the Expr data type, this is represented as the Term tree: Note how the outer parentheses are represented as the separators on the paren tree. As another example, consider the AST node of 1+2, which is add(lit(1), lit(2)).…”

“…We have modified the Rascal pattern matching engine to support matching against SSTs, both using abstract patterns and concrete syntax patterns (using the Concretely framework [1]). Listing 7 shows an example transformation that adds a logger argument as the final formal parameter to function definitions, this time using abstract matching.…”

“…While this allows for highly specific pattern specification, it requires the meta programmer to have in-depth knowledge of the intricacies of the abstract syntax. Concretely is a technique that allows specification of patterns in concrete syntax, converting these to abstract patterns under the hood [1]. We have adapted Concretely to produce SSTs, allowing us to write concrete syntax both in left-hand side and right-hand side patterns of rewrite rules.…”

“…In Section 4.2 we will illustrate how this implementation allows rewriting similar to the rules of HifiTRS, using an example to add a logger parameter as the final formal argument to function definitions. Finally, in Section 4.3, we will discuss our efforts of implementing SSTs for C++ in Rascal; we will show how to implement the Encapsulate Field refactoring using our SST implementation, and additionally show that we can write the left-hand side of a rewrite rule in concrete syntax as well, by employing the Concretely framework [1].…”