A Framework for the Construction of Upper Bounds on the Number of Affine Linear Regions of ReLU Feed-Forward Neural Networks

Hinz, Peter; Geer, Sara van de

doi:10.1109/tit.2019.2927252

Cited by 15 publications

(33 citation statements)

References 2 publications

(1 reference statement)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Affine linear regions can be defined as the connected components of R N0 \ H, where H is the set of non-differentiability of the realization 20 Φ (N, R ) (•, θ). A refined analysis on the number of such regions was, for example, conducted by [HvdG19]. It is found that deep ReLU neural networks can exhibit significantly more regions than their shallow counterparts.…”

Section: Alternative Notions Of Expressivitymentioning

confidence: 99%

The Modern Mathematics of Deep Learning

Berner¹,

Kutyniok²,

Petersen³

2021

Preprint

View full text Add to dashboard Cite

We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical framework of learning theory. These questions concern: the outstanding generalization power of overparametrized neural networks, the role of depth in deep architectures, the apparent absence of the curse of dimensionality, the surprisingly successful optimization performance despite the non-convexity of the problem, understanding what features are learned, why deep architectures perform exceptionally well in physical problems, and which fine aspects of an architecture affect the behavior of a learning task in which way. We present an overview of modern approaches that yield partial answers to these questions. For selected approaches, we describe the main ideas in more detail. * This review paper will appear as a book chapter in the book "Theory of Deep Learning" by Cambridge University Press.

show abstract

Section: Alternative Notions Of Expressivitymentioning

confidence: 99%

The Modern Mathematics of Deep Learning

Berner¹,

Kutyniok²,

Petersen³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Works in this direction include [45,46,11,47,52,32,36] and earlier works for Boolean circuits and sum-product networks [20,21,10]. The number of linear regions of the functions represented by networks with piecewise linear activations has sparked substantial interest in the study of neural networks, with works including [34,47,33,5,40,23]. Recent works have explored approaches based on tropical geometry [55,9,3] and power diagram subdivisions [6], while others have studied the expectated number of linear regions for typical choices of the parameters in the case of ReLU networks [18,19], empirical enumeration [39], and the relations between linear regions and the behavior of algorithms that are used to select the parameters of neural networks based on data, such as speed of convergence and implicit biases of gradient descent [44,56,26].…”

Section: Introductionmentioning

confidence: 99%

Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums

Montúfar¹,

Ren²,

Zhang³

2021

Preprint

View full text Add to dashboard Cite

We present results on the number of linear regions of the functions that can be represented by artificial feedforward neural networks with maxout units. A rank-k maxout unit is a function computing the maximum of k linear functions. For networks with a single layer of maxout units, the linear regions correspond to the upper vertices of a Minkowski sum of polytopes. We obtain face counting formulas in terms of the intersection posets of tropical hypersurfaces or the number of upper faces of partial Minkowski sums, along with explicit sharp upper bounds for the number of regions for any input dimension, any number of units, and any ranks, in the cases with and without biases. Based on these results we also obtain asymptotically sharp upper bounds for networks with multiple layers.

show abstract

“…These are helpful for the improvement of network structure in this paper. Thanks to the nonlinear characteristics of activation function, neural network with improved activation function has shown good results [18][19][20][21][22][23][24]. In order to learn the distribution characteristics of the nonlinear data better, some improvements of the network are mainly focused on the network's depth.…”

Section: Introductionmentioning

confidence: 99%

An Automatic Garbage Classification System Based on Deep Learning

Kang

Yang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Garbage classification has always been an important issue in environmental protection, resource recycling and social livelihood. In order to improve the efficiency of front-end garbage collection, an automatic garbage classification system is proposed based on deep learning. Firstly, the overall system of the garbage bin is designed, including the hardware structure and the mobile app. Secondly, the proposed garbage classification algorithm is based on ResNet-34 algorithm, and its network structure is further optimized by three aspects, including the multi feature fusion of input images, the feature reuse of the residual unit, and the design of a new activation function. Finally, the superiority of the proposed classification algorithm is verified with the constructed garbage data. The classification accuracy of the proposed algorithm is enhanced by 1.01%. The experimental results show that the classification accuracy is as high as 99%, the classification cycle of the system is as quick as 0.95 s.

show abstract

A Framework for the Construction of Upper Bounds on the Number of Affine Linear Regions of ReLU Feed-Forward Neural Networks

Cited by 15 publications

References 2 publications

The Modern Mathematics of Deep Learning

The Modern Mathematics of Deep Learning

Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums

An Automatic Garbage Classification System Based on Deep Learning

Contact Info

Product

Resources

About