Describing Console I/O Behavior for Testing Student Submissions in Haskell

Abstract

We present a small, formal language for specifying the behavior of simple console I/O programs. The design is driven by the concrete application case of testing interactive Haskell programs written by students. Specifications are structurally similar to lexical analysis regular expressions, but are augmented with features like global variables that track state and history of program runs, enabling expression of an interesting range of dynamic behavior. We give a semantics for our specification language based on acceptance of execution traces. From this semantics we derive a definition of the set of all traces valid for a given specification. Sampling that set enables us to mechanically check program behavior against specifications in a probabilistic fashion. Beyond testing, other possible uses of the specification language in an education context include related activities like providing more helpful feedback, generating sample solutions, and even generating random exercise tasks.

A Framework for Generating Diverse Haskell-IO Exercise Tasks

“…For convenience we define some combinators for defining templates and instances that make these definitions more directly readable. Before diving into the description of I/O tasks, we give a short summary of the specification language, introduced in our previous work [9], as it forms the basis of the presented approach. Specification expressed in the language describe program behavior in terms of the traces, i.e, sequences of read and written values, a program should produce.…”

“…We will not present the language in the formal way of [9], but instead use the Haskell embedding, since this embedding is what we are going to use. Specifications are built from three atomic specifications, a branching and a loop construct and sequential composition.…”

“…The testing approach of the specification framework relies on programs being represented in an inspectable version of the standard Haskell IO monad [9,6].…”

“…Using fulfills, which is provided by the implementation of [9], we can construct our first I/O specific requirement for a correct task solution.…”

“…We have recently designed and implemented a language for the specification of console I/O programs [9][10] that allows the formulation of desired I/O behavior and probabilistic testing of actual programs, written in Haskell, against the specified behavior. We built this language to bring our testing capabilities of exercises on Haskell I/O more in line with how one can test exercises on pure programs using, for example, QuickCheck [1].…”