Recovering grammar relationships for the Java Language Specification

Abstract

Grammar convergence is a method that helps discovering relationships between different grammars of the same language or different language versions. The key element of the method is the operational, transformation-based representation of those relationships. Given input grammars for convergence, they are transformed until they are structurally equal. The transformations are composed from primitive operators; properties of these operators and the composed chains provide quantitative and qualitative insight into the relationships between the grammars at hand.We describe a refined method for grammar convergence, and we use it in a major study, where we recover the relationships between all the grammars that occur in the different versions of the Java Language Specification (JLS). The relationships are represented as grammar transformation chains that capture all accidental or intended differences between the JLS grammars. This method is mechanized and driven by nominal and structural differences between pairs of grammars that are subject to asymmetric, binary convergence steps.We present the underlying operator suite for grammar transformation in detail, and we illustrate the suite with many examples of transformations on the JLS grammars. We also describe the extraction effort, which was needed to make the JLS grammars amenable to automated processing. We include substantial metadata about the convergence process for the JLS so that the effort becomes reproducible and transparent.

Negotiated Grammar Evolution.

“…In XBGF, any change to a grammar can be expressed as a chain of calls to grammar transformation operators from an extensive operator suite [LZ11]. Since the activity of creating such a chain closely resembles programming, it is commonly referred to as "grammar programming" [DCMS02], "metaprogramming" [KvdSV09] or "grammar engineering" [LV01,KLV05], even though all three terms also cover other activities that go beyond operator-based manipulation of grammars in a broad sense in the style of event sourcing [Fow05].…”

“…Same as above, but the choice is made automatically, based on a weighting algorithm or even at random, so the process continues as long as there is at least one successful alternative for each step. This option corresponds to heuristic-based automation methods, in particular for error recovery and robust processing [LZ11].…”

“…Among operands, a, b, n are nonterminals, p, q are production rules, x, y are grammatical expressions, m are markers. A "preserving" transformation preserves the language defined by the grammar, an "increasing" or "decreasing" one makes it larger or smaller, and a "revising" operator can have other outcomes [LZ11]. An "updative" transformation preserves the size of the modelling space, an "additive" or "subtractive" transformation makes it larger or smaller [CREP08].…”

“…In previous work on XBGF [LZ09,LZ11,...] we have differentiated between the following groups of transformation operators:…”

Notation-parametric grammar recovery

“…With many grammars in existence, it is desirable for a language engineer to reuse them instead of developing new ones by hand from scratch. Quite a number of grammar recovery projects were attempted and successfully performed during the last decade and a half [14,15,17,20,21,23,24,25,26,28,29]. When recovering a language grammar from an existing source, one faces various challenges ranging from character level issues (e.g., layout inconsistencies) to language level issues (e.g., grammar connectedness).…”