Narcissus: correct-by-construction derivation of decoders and encoders from binary formats

Abstract

It is a neat result from functional programming that libraries of parser combinators can support rapid construction of decoders for quite a range of formats. With a little more work, the same combinator program can denote both a decoder and an encoder. Unfortunately, the real world is full of gnarly formats, as with the packet formats that make up the standard Internet protocol stack. Most past parser-combinator approaches cannot handle these formats, and the few exceptions require redundancy ś one part of the natural grammar needs to be hand-translated into hints in multiple parts of a parser program. We show how to recover very natural and nonredundant format specifications, covering all popular network packet formats and generating both decoders and encoders automatically. The catch is that we use the Coq proof assistant to derive both kinds of artifacts using tactics, automatically, in a way that guarantees that they form inverses of each other. We used our approach to reimplement packet processing for a full Internet protocol stack, inserting our replacement into the OCaml-based MirageOS unikernel, resulting in minimal performance degradation. ] that aim to reduce opportunities for user error in writing encoders and decoders, but these systems are quite tricky to get right and have themselves been sources of serious security bugs [CVE 2016].Combinator libraries are an alternative approach to the rapid development of parsers which has proven particularly popular in the functional-programming community [Leijen and Meijer 2001]. This approach has been adapted to generate both parsers and pretty printers from single programs [Kennedy 2004;Rendel and Ostermann 2010]. Unfortunately, unverified combinator libraries suffer from the same potential for bugs as code-generation frameworks, with the additional possibility for users to introduce errors when extending the library with new combinators. This paper presents Narcissus, a combinator-style framework for the Coq proof assistant that eliminates the possibility of such bugs, enabling the derivation of encoders and decoders that are correct by construction. Each derived encoder and decoder is backed by a machine-checked functionalcorrectness proof, and Narcissus leverages Coq's proof automation to help automate both the construction of encoders and decoders and their correctness proofs. Key to our approach is how it propagates information through a derivation, in order to generate decoders and encoders for the sorts of non-context-free languages that often appear in standard networking protocols.We begin by introducing the key features of Narcissus with a series of increasingly complex examples, leading to a hypothetical format of packets sent by a temperature sensor to a smart home controller. In order to build up the reader's intuition, we deliberately delay a discussion of the full details of our approach until Section 2. The code accompanying our tour is included in the Narcissus repository 1 in the src/Narcissus/Examples/README.v file, and can be run in Coq ver...

Investigating Automatic Code Generation for Network Packet Parsing

“…These include Nail [34], which builds upon Hammer [35], and produces parser combinators for grammars specified in domainspecific languages, using stream transformations and arenas to support complex protocols. Narcissus [36] supports multi-stage parsing with a framework for the Coq proof assistant.…”

Parsing Protocol Standards to Parse Standard Protocols

“…Approaches are starting to be developed that give stronger type guarantees. Narcissus [8] is a parser combinator-style framework for the Coq proof assistant [9], enabling derivation of veriable encoders and decoders. Beyond likely usability barriers to the adoption of a framework based on a non-mainstream tool such as Coq, which we turn to in Section 4, Narcissus does not provide support for persistent storage throughout the parsing process.…”

A verified packrat parser interpreter for parsing expression grammars

“…6 This can easily be replaced by a notion of end of stream character if the input bound is unknown. 8. star (e, s,T 0 ,T s ) with T 0 S. To keep the star structure, we ask forT s to be either of type star or skip.…”

“…Usually, as we mentioned and as in [11] and [15], the parsing is defined as a relationship between the inputs and the outputs that verifies a certain number of derivation rules corresponding to all the parsing cases. Lemmas that link the properties of the grammar to the parsing relationship 8 8 Those lemmas are the following : (1) If the parsing fails, the grammar is of type P ⊥ (2) If the parsing succeed without consuming any input, the grammar is of type P 0 (3) If the parsing succeed consuming at least one token, the grammar is of type P >0 . are then proved to ensure that any algorithm following the parsing rules would terminate.…”