Morbig: A Static parser for POSIX shell

Abstract

The POSIX shell language defies conventional wisdom of compiler construction on several levels: The shell language was not designed for static parsing, but with an intertwining of syntactic analysis and execution by expansion in mind. Token recognition cannot be specified by regular expressions, lexical analysis depends on the parsing context and the evaluation context, and the shell grammar given in the specification is ambiguous. Besides, the unorthodox design choices of the shell language fit badly in the usual specification languages used to describe other programming languages. This makes the standard usage of Lex and Yacc as a pipeline inadequate for the implementation of a parser for POSIX shell. The existing implementations of shell parsers are complex and use low-level character-level parsing code which is difficult to relate to the POSIX specification. We find it hard to trust such parsers, especially when using them for writing automatic verification tools for shell scripts.This paper offers an overview of the technical difficulties related to the syntactic analysis of the POSIX shell language. It also describes how we have resolved these difficulties using advanced parsing techniques (namely speculative parsing, parser state introspection, context-dependent lexical analysis and longest-prefix parsing) while keeping the implementation at a sufficiently high level of abstraction so that experts can check that the POSIX standard is respected. The resulting tool, called Morbig, is an open-source static parser for a well-defined and realistic subset of the POSIX shell language.

The CoLiS platform for the analysis of maintainer scripts in Debian software packages

“…This study had several additional outcomes. The toolchain includes tools for parsing and light static analysis of shell scripts [36], an engine for the symbolic execution of imperative languages based on first-order logics representation of program configurations [7], and an efficient decision procedure for feature tree logic. We also provide a formal specification of Posix commands used in Debian scripts in terms of a first-order logic [28].…”

“…The morbig parser was designed by some of the authors in 2018 [36]. Its implementation aims at employing code generators as much as possible to keep the implementation at a high level of abstraction, to simplify its maintenance, and to ease checking of compliance with the Posix standard.…”

On Parsing Programming Languages with Turing-Complete Parser

“…Another way is using context-free parsers extended with a subset of approaches used for parsing specific types of non-context-free language constructs and features. [25]. One such approach is stateful parsing, which maintains a global parse-wide state.…”

RNGSGLR: Generalization of the Context-Aware Scanning Architecture for All Character-Level Context-Free Languages

“…Legacy programming languages, such as COBOL and Fortran. Additionally, POSIX shell [21] could be supported as well (with a few extensions for the syntax that is not context-free). These languages were developed before the advent of modern parsing methods.…”