On Correctness of Automatic Differentiation for Non-Differentiable Functions

Abstract

Differentiation lies at the core of many machine-learning algorithms, and is wellsupported by popular autodiff systems, such as TensorFlow and PyTorch. Originally, these systems have been developed to compute derivatives of differentiable functions, but in practice, they are commonly applied to functions with non-differentiabilities. For instance, neural networks using ReLU define nondifferentiable functions in general, but the gradients of losses involving those functions are computed using autodiff systems in practice. This status quo raises a natural question: are autodiff systems correct in any formal sense when they are applied to such non-differentiable functions? In this paper, we provide a positive answer to this question. Using counterexamples, we first point out flaws in oftenused informal arguments, such as: non-differentiabilities arising in deep learning do not cause any issues because they form a measure-zero set. We then investigate a class of functions, called PAP functions, that includes nearly all (possibly nondifferentiable) functions in deep learning nowadays. For these PAP functions, we propose a new type of derivatives, called intensional derivatives, and prove that these derivatives always exist and coincide with standard derivatives for almost all inputs. We also show that these intensional derivatives are what most autodiff systems compute or try to compute essentially. In this way, we formally establish the correctness of autodiff systems applied to non-differentiable functions.Preprint. Under review.

Towards Denotational Semantics of AD for Higher-Order, Recursive, Probabilistic Languages

“…• If AD is applied to a recursive program, does the derivative halt on the same inputs as the original program? We show that the answer is yes, and that restricted to this common domain of definition, AD computes an intensional derivative of the input program [Lee et al, 2020].…”

“…Semantic Framework Piecewise Recursion Higher-Order Approach AD PPL Huot et al [2020] Denotational Vákár [2020] Denotational Lee et al [2020] Denotational Abadi and Plotkin [2020] Both Mazza and Pagani [2021] Operational Mak et al [2021] Operational Ours Denotational…”

“…Our approach is inspired by Lee et al [2020], who provided a similar characterization of AD for a first-order language with branching (but not recursion). Section 2 reviews their development, and gives intuition for why recursion, higher-order functions, and partiality present additional challenges.…”

“…In Section 3, we give our solution to these challenges, culminating in a similar result to that of Lee et al [2020] but for a more expressive language. We recover as a straightforward corollary the result of Mazza and Pagani [2021] that when applied to recursive, higher-order programs, AD fails at a measure-zero set of inputs.…”

“…The growing importance of AD for learning and inference has inspired a torrent of work on the semantics of differentiable languages, summarized in Table 1. We build on existing denotational approaches, particularly those of Huot et al [2020] and Vákár [2020], but incorporate ideas from Lee et al [2020] to handle piecewise functions, and Vákár et al [2019] to model probabilistic programs. Mazza and Pagani [2021] consider a language equally expressive as our deterministic fragment; we give a novel and complementary denotational account (their approach is operational).…”