Log-Sum-Exp Neural Networks and Posynomial Models for Convex and Log-Log-Convex Data

Abstract

We show in this paper that a one-layer feedforward neural network with exponential activation functions in the inner layer and logarithmic activation in the output neuron is an universal approximator of convex functions. Such a network represents a family of scaled log-sum exponential functions, here named LSET . Under a suitable exponential transformation, the class of LSET functions maps to a family of generalized posynomials GPOST , which we similarly show to be universal approximators for log-log-convex functions.A key feature of an LSET network is that, once it is trained on data, the resulting model is convex in the variables, which makes it readily amenable to efficient design based on convex optimization. Similarly, once a GPOST model is trained on data, it yields a posynomial model that can be efficiently optimized with respect to its variables by using geometric programming (GP).The proposed methodology is illustrated by two numerical examples, in which, first, models are constructed from simulation data of the two physical processes (namely, the level of vibration in a vehicle suspension system, and the peak power generated by the combustion of propane), and then optimization-based design is performed on these models.

A Universal Approximation Result for Difference of Log-Sum-Exp Neural Networks

“…The following preliminary definitions are instrumental for our purposes: a subset R ⊂ R n >0 will be said to be log-convex if its image by the map that takes the logarithm entry-wise is convex. We shall say that a function ψ T ∈ GPOS T has rational parameters if it can be written as in (7) with ψ given by (6), in such a way that T , the entries of the vectors α (1) , . .…”

“…Similar to our previous work [7] on which it builds, the present paper is inspired by ideas from tropical geometry and max-plus algebra. The class of functions in LSE that we study here plays a key role in Viro's patchworking methods [9], [10] for real curves.…”

“…The maps in LSE T are special instances of the log-Laplace transforms of nonnegative measures, studied in [17]. In particular, the maps in LSE T are smooth, they are convex (this is an easy consequence of Cauchy-Schwarz inequality), and they are even strictly convex if the vectors − → α (k) constitute an affine generating family of R n , see [7,Prop. 1].…”

“…We denote by POS the class of functions ψ : R n >0 → R >0 of the form (6). Posynomials are log-log-convex functions, meaning that the log of a posynomial ψ is convex in the log of its argument, see, e.g., Section II.B of [7]. We denote by GPOS T the class of functions that can be expressed as…”

“…A. Preliminary: approximation of convex functions We start by recalling a key result of [7], stating that functions in LSE T are universal smooth approximators of convex functions, see Theorem 2 in [7].…”