High-fidelity metaprogramming with separator syntax trees

Abstract

Many metaprogramming tasks, such as refactorings, automated bug fixing, or large-scale software renovation, require high-fidelity source code transformations-transformations which preserve comments and layout as much as possible. Abstract syntax trees (ASTs) typically abstract from such details, and hence would require pretty printing, destroying the original program layout. Concrete syntax trees (CSTs) preserve all layout information, but transformation systems or parsers that support CSTs are rare and can be cumbersome to use. In this paper we present separator syntax trees (SSTs), a lightweight syntax tree format, that sits between AST and CSTs, in terms of the amount of information they preserve. SSTs extend ASTs by recording textual layout information separating AST nodes. This information can be used to reconstruct the textual code after parsing, but can largely be ignored when implementing high-fidelity transformations. We have implemented SSTs in Rascal, and show how it enables the concise definition of high-fidelity source code transformations using a simple refactoring for C++. CCS Concepts • Software and its engineering → Translator writing systems and compiler generators; Source code generation; Software maintenance tools.

Large‐scale semi‐automated migration of legacy C/C++ test code

“…Another way of circumventing this tangling of code changes with their implementation as edit scripts would have been to use Rascal 's experimental concrete syntax features for abstract syntax trees 11 . This would allow us to write both the patterns and their substitutions in the C++ source language, while the same ASTs would be used under the hood.…”

“…Augmenting the available functionality for C++ with concrete syntax pattern, for instance by implementing the Concretely framework, 12 would certainly make our patterns more concise and more readable. Furthermore, this would fully eliminate the bookkeeping of applied code changes, including file offset corrections, as (changed) parse trees could simply be unparsed 11 …”

“…Another way of circumventing this tangling of code changes with their implementation as edit scripts would have been to use Rascal's experimental concrete syntax features for abstract syntax trees. 11 This would allow us to write both the patterns and their substitutions in the C++ source language, while the same ASTs would be used under the hood. This would be a readable solution and the edit scripts would be derivable without the need for a pretty-printer; in fact, this would even allow us to directly unparse modified syntax trees.…”

“…Furthermore, this would fully eliminate the bookkeeping of applied code changes, including file offset corrections, as (changed) parse trees could simply be unparsed. 11 One of our key observations is that when conducting a significant refactoring task such as the one described in Section 2, it is imperative to be flexible. In this particular case, for instance, the initial assumption was that only C++ files would be affected, and hence, being able to parse and analyze C++ files would suffice.…”