Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums

Abstract

We present results on the number of linear regions of the functions that can be represented by artificial feedforward neural networks with maxout units. A rank-k maxout unit is a function computing the maximum of k linear functions. For networks with a single layer of maxout units, the linear regions correspond to the upper vertices of a Minkowski sum of polytopes. We obtain face counting formulas in terms of the intersection posets of tropical hypersurfaces or the number of upper faces of partial Minkowski sums, along with explicit sharp upper bounds for the number of regions for any input dimension, any number of units, and any ranks, in the cases with and without biases. Based on these results we also obtain asymptotically sharp upper bounds for networks with multiple layers.

Towards Lower Bounds on the Depth of ReLU Neural Networks

“…In fact, there is a one-to-one correspondence between elements of CCPWL n and Newt n , which is nicely compatible with some (functional and polyhedral) operations. This correspondence has been studied before in the tropical geometry [Maclagan andSturmfels, 2015, Joswig, 2022], convex geometry 1 [Hiriart-Urruty and Lemaréchal, 1993], as well as neural network literature [Zhang et al, 2018, Charisopoulos and Maragos, 2018, Alfarra et al, 2020, Montúfar et al, 2021. We summarize the key findings of this correspondence relevant to our work in the following proposition: Proposition 4.5.…”

“…The purpose of this section is to prove that for fixed dimension n, the required width for exact, depth-minimal representation of a CPWL function can be polynomially bounded in the number p of affine pieces; in particular p O(n 2 ) . This is closely related to works that bound the number of linear pieces of an NN as a function of the size , Raghu et al, 2017, Montúfar et al, 2021. It can also be seen as a counterpart, in the context of exact representations, to quantitative universal approximation theorems that bound the number of neurons required to achieve a certain approximation guarantee; see, e.g., Barron [1993Barron [ , 1994, Mhaskar [1993], Pinkus [1999], Mhaskar [1996], Mhaskar and Micchelli [1995].…”

“…Linear regions of these functions correspond to vertices of so-called Newton polytopes associated with these tropical polynomials. Applications of this correspondence include bounding the number of linear regions of a neural network [Zhang et al, 2018, Charisopoulos and Maragos, 2018, Montúfar et al, 2021 and understanding decision boundaries [Alfarra et al, 2020]. In Section 4 we present a novel application of tropical concepts to understand neural networks.…”

Training Fully Connected Neural Networks is $\exists\mathbb{R}$-Complete

“…However, deeper networks require much less neurons to reach the same expressive power, yielding a potential theoretical explanation of the dominance of deep networks in practice [7,29,42,44,53,62,65,68,79,80,83]. Other related work includes counting and bounding the number of linear regions [43,59,60,64,65,74], classifying the set of functions exactly representable by different architectures [7,23,46,47,61,86], or analyzing the memorization capacity of ReLU networks [82,84,85].…”

Tropical reproducing kernels and optimization

“…Several research directions have been explored at the interface between tropical geometry, probablity theory and machine learning. These include studies of the tropicalization of stochastic processes (Akian et al, 1994) or of Gaussian measures (Tran, 2020), tropical support vector machines (Yoshida et al, 2021), tropical principal component analysis (Yoshida et al, 2019) inspired by phylogenetic studies, quantification of the expressivity of deep neural networks (Zhang et al, 2018;Montúfar et al, 2021) or their approximation (Calafiore et al, 2020) through tropical methods. A survey of some of these approaches can be found in Maragos et al (2021).…”