A Language for Differentiable Functions

Abstract

Abstract-We introduce a typed lambda calculus in which real numbers, real functions, and in particular continuously differentiable and more generally Lipschitz functions can be defined. Given an expression representing a real-valued function of a real variable in this calculus, we are able to evaluate the expression on an argument but also evaluate the generalised derivative, i.e., the L-derivative, equivalently the Clarke gradient, of the expression on an argument. The language is an extension of PCF with a real number data-type, similar to Real PCF and RL, but is equipped with primitives for min and weighted average to capture computable continuously differentiable or Lipschitz functions on real numbers. We present an operational semantics and a denotational semantics based on continuous Scott domains and several logical relations on these domains. We then prove an adequacy result for the two semantics. The denotational semantics is closely linked with Automatic Differentiation also called Algorithmic Differentiation, which has been an active area of research in numerical analysis for decades, and our framework can also be considered as providing denotational semantics for Automatic Differentiation. We derive a definability result showing that for any computable Lipschitz function there is a closed term in the language whose evaluation on any real number coincides with the value of the function and whose derivative expression also evaluates on the argument to the value of the generalised derivative of the function.

A simple differentiable programming language

“…Another interesting possibility would be to work with non-differentiable functions like ReLU or with non-smooth functions. For the former, one might use Clarke sub-gradients [Clarke 1990], following [Di Gianantonio and Edalat 2013] (the Clarke sub-gradient of ReLU at 0 is the interval [0, 1]); for the latter, one may use C k functions. Yet another possibility would be to work with approximate reals, rather than reals, and to seek numerical accuracy theorems; one might employ a domain-theoretic notion of sub-differentiation of functions over Scott's interval domain, generalizing the Clarke sub-gradient (see [Edalat and Lieutier 2004;Edalat and Maleki 2018]).…”

“…Our work is also related to important papers by Ehrhard, Regnier, et al [Ehrhard and Regnier 2003], and by Di Gianantonio and Edalat [Di Gianantonio and Edalat 2013]. Ehrhard and Regnier introduce the differential λ-calculus; this is a simply-typed higher-order λ-calculus with a forwardmode differentiation construct which can be applied to functions of any type.…”

Polynomial Time over the Reals with Parsimony

“…[1][2][3][4]. Several programming languages studying mathematical properties of real functions such as computable functions [5,6], Lipschitz-functions [7], or analytical functions [8] have been introduced and studied but no programming language characterizing polynomial time over the reals has emerged. A programming language based characterization of polynomial time over the reals would be highly valuable as it would provide a programmer the opportunity to write libraries of efficient programs where any computation can be approximated in feasible time in the output precision.…”

Differential Calculus with Imprecise Input and Its Logical Framework

“…In [7], a typed lambda calculus in the framework of an extension of Real PCF [6,17,22] was introduced in which in particular continuously differentiable and more generally Lipschitz functions can be defined. Given an expression representing a real-valued function of a real variable in this language, one is able to evaluate the expression on an argument, representing an interval, but also evaluate the generalised derivative, i.e., the L-derivative, equivalently the Clarke gradient, of the expression on an interval.…”