Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

Abstract

Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave, Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable sets, our model provides variational formulas for nonsmooth automatic differentiation oracles, as for instance the famous backpropagation algorithm in deep learning. Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.

Incremental Without Replacement Sampling in Nonconvex Optimization

“…, n}, and moving in the direction of the gradient of F corrupted by centered noise inherent to subsampling. This allows to study such algorithms in the broader context of stochastic approximation, initiated by Robins and Monro [51] with many subsequent works [37,6,33,17,42,23,14].…”

“…Nonsmooth setting: f i are path differentiable. Such functions constitute a subclass of Lipschitz functions which enjoy some of the nice properties of nonsmooth convex functions [14] in particular, an operational chain rule [23]. In this case, d i is a selection in a conservative field for f i .…”

“…In this case, d i is a selection in a conservative field for f i . Examples of such objects include convex subgradients if f i is convex, the Clarke subgradient, which is an extension to the nonconvex setting, as well as the output of automatic differentiation applied to a nonsmooth program, see [14] for more details.…”

“…In the nonsmooth world the backpropagation algorithm [53] used in deep learning suffers from inconsistent behaviors and may not provide subgradient of any kind [31,32]. We use the recently introduced tool of conservative fields and path differentiable function [14] capturing the full complexity of backpropagation oracles. Our proof essentially relies on the notion of Asymptotic Pseudo Trajectory (APT) for differential inclusions [5,7].…”

“…We need to resort to a continuous extension of the right hand side, which becomes set valued, providing a weaker notion of solution. We use the recently introduced notion of conservativity [14] which captures the complexity of automatic differentiation oracles in nonsmooth settings [15]. Recall that the set valued map D is conservative for the locally Lipschitz function f , if it has a closed graph and for any locally Lipschitz curve…”