How to evaluate the performance of gradual type systems

Abstract

A sound gradual type system ensures that untyped components of a program can never break the guarantees of statically typed components. This assurance relies on runtime checks, which in turn impose performance overhead in proportion to the frequency and nature of interaction between typed and untyped components. The literature on gradual typing lacks rigorous descriptions of methods for measuring the performance of gradual type systems. This gap has consequences for the implementors of gradual type systems and developers who use such systems. Without systematic evaluation of mixed-typed programs, implementors cannot precisely determine how improvements to a gradual type system affect performance. Developers cannot predict whether adding types to part of a program will significantly degrade (or improve) its performance. This paper presents the first method for evaluating the performance of sound gradual type systems. The method quantifies both the absolute performance of a gradual type system and the relative performance of two implementations of the same gradual type system. To validate the method, the paper reports on its application to 20 programs and 3 implementations of Typed Racket.

Which of my transient type checks are not (almost) free?

“…That is, for these benchmarks on transient type checks, measuring just the untyped and fullytyped configurations would provide excellent estimates of a benchmark's performance bounds. This is different to the experience of other kinds of gradual typing, where the best, and most importantly worst, configurations are not always those fully typed or fully untyped [16]. However, the Richards benchmark does have sections outside these bounds, and isolated executions of a couple of others (Fannkuch and DeltaBlue) are also outliers.…”

“…Our goal is to identify which type annotations in Grace programs cause performance effects. To this end, we built upon the so-called "Takikawa" or "Takikawa-Greenman" evaluation protocol [16,36]. It uses 2 N configurations of each benchmark.…”

“…This has also lead to a technique to evaluate the performance of gradual typing, the "Takikawa" protocol [16,36,38]. The Takikawa protocol was created to measure the run-time cost of gradual typing by testing various configurations of typed and untyped code.…”

Complete monitors for gradual types

“…First, Natural is best for developers who wish to get precise blame information, and it obviously suffers from high overhead. As Greenman et al [2019b] point out, the latter is partly due to the wrappers needed to protect typed values from bad untyped code and untyped values from bad types.…”

“…Second, these checks must enforce the behaviors allowed by the types. The implementation of complete monitoring demands a mechanism for tracking types, something that is occasionally impossible [Vitousek et al 2017] and always expensive [Allende et al 2013;Greenman et al 2019b;Takikawa et al 2015]. Studying typed-untyped interaction from the perspective of complete monitoring, though, suggests weaker properties and a compromise.…”