The Racket virtual machine and randomized testing

Abstract

We present a PLT Redex model of a substantial portion of the Racket virtual machine and bytecode verifier (formerly known as MzScheme), along with lessons learned in developing the model. Inspired by the "warts-and-all" approach of the VLISP project, in which Wand et al. produced a verified implementation of Scheme, our model reflects many of the realities of a production system. Our methodology departs from the VLISP project's in its approach to validation; instead of producing a proof of correctness, we explore the use of QuickCheck-style randomized testing, finding it a cheap and effective technique for discovering a variety of errors in the model-from simple typos to more fundamental design mistakes.

Making Random Judgments: Automatically Generating Well-Typed Terms from the Definition of a Type-System

“…The models in the benchmark come from a number of different sources, some synthesized based on our experience for the benchmark, and some drawn from outside sources or pre-existing efforts in Redex. The latter are based on Appel et al (2012)'s list machine benchmark, the model of contracts for delimited continuations developed by Takikawa et al (2013), and the model of the Racket virtual machine from Klein et al (2013). Detailed descriptions of all the models and bugs in the benchmark can be found in Findler et al (2014).…”

“…Until recently, Redex used a naive generation strategy: it simply randomly picks productions from the grammar of the language to build a term and then checks to see if that falsifies the property of interest. For untyped models, or when the model author writes a "fixing" function that makes expressions more likely to type-check (e.g., by writing a post-processing function that binds free variables), this naive technique is effective (Klein 2009;Klein et al 2012;Klein et al 2013). With typed models, however, such randomly generated terms rarely type check and so the testing process spends most of its time rejecting ill-typed terms instead of actually testing the model.…”

Experience report

“…The functional programming community has a successful history of using random testing (Claessen and Hughes 2000) to find bugs in programs ranging from undergraduate coursework (Page et al 2008) to full-scale language models (Berghofer and Nipkow 2004;Klein et al 2012;Klein et al 2013). Inspired by these success stories, we decided to investigate whether random testing would help us detect bugs in the Typed Racket base environmentspecifically in the implementation of its numeric tower.…”

“…Redex's random testing facility, redex-check, takes as input a language grammar and a property expressed about terms in that language, generates random terms and checks whether the desired property holds. It has been previously used to find mistakes in nine ICFP 2009 papers (Klein et al 2012) and to find bugs in a model of the Racket virtual machine (Klein et al 2013).…”

A Semantics for Context-Sensitive Reduction Semantics

“…Specifically, Redex supports rapid prototyping of context-sensitive operational semantics, random testing, automatic typesetting, and, via its embedding in Racket, access to a large palette of standard programming tools. Redex is widely used, having supported several dozen research papers as well as the latest Scheme standard (Sperber et al 2007) and a number of larger models, including one of the Racket virtual machine (Klein et al 2010).…”