Empirical Bounds on Linear Regions of Deep Rectifier Networks

Serra, Thiago; Ramalingam, Srikumar

doi:10.1609/aaai.v34i04.6016

Cited by 32 publications

(43 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, enumerating solutions of a mixed-integer program can be computationally intractable. To address this issue, a probabilistic algorithm was proposed in [97] to produce lower bounds to the number of possible solutions.…”

Section: ) Mixed-integer Formulationsmentioning

confidence: 99%

“…, aK x + bK y + cK ) . (97) This consists of K planes of unknown slopes estimated by using K-means on the numerical gradients of the 2-D data, whereas the intercepts c k are computed using the tropical fitting algorithm as in (93), which solves the unconstrained ∞ problem. In this combined approach, the first step (K-means) is heuristic, yielding probably a local minimum for the slope estimation subproblem, whereas the second step (tropical regression for the intercepts) yields a global minimum optimally solving the unconstrained ∞ problem.…”

Section: Optimally Fitting Tropical Polynomial Curves and Surfacesmentioning

confidence: 99%

See 1 more Smart Citation

Tropical Geometry and Machine Learning

2021

View full text Add to dashboard Cite

show abstract

Section: ) Mixed-integer Formulationsmentioning

confidence: 99%

Section: Optimally Fitting Tropical Polynomial Curves and Surfacesmentioning

confidence: 99%

Tropical Geometry and Machine Learning

2021

View full text Add to dashboard Cite

show abstract

“…Modeling Optimization Applications Involving Neural Network Surrogates Related Work. Prior approaches to optimization over neural networks using MIP formulations include [8,15,16,28,47,51]. These approaches primarily model the piecewise ReLU constraint using standard big-M modeling tricks.…”

Section: Outline and Contributionsmentioning

confidence: 99%

Modeling Design and Control Problems Involving Neural Network Surrogates

Yang¹,

Balaprakash²,

Leyffer³

2021

Preprint

View full text Add to dashboard Cite

We consider nonlinear optimization problems that involve surrogate models represented by neural networks. We demonstrate first how to directly embed neural network evaluation into optimization models, highlight a difficulty with this approach that can prevent convergence, and then characterize stationarity of such models. We then present two alternative formulations of these problems in the specific case of feedforward neural networks with ReLU activation: as a mixed-integer optimization problem and as a mathematical program with complementarity constraints. For the latter formulation we prove that stationarity at a point for this problem corresponds to stationarity of the embedded formulation. Each of these formulations may be solved with state-of-the-art optimization methods, and we show how to obtain good initial feasible solutions for these methods. We compare our formulations on three practical applications arising in the design and control of combustion engines, in the generation of adversarial attacks on classifier networks, and in the determination of optimal flows in an oil well network.

show abstract

“…This way, equivalent neural networks of smaller size can be obtained. The computation of linear regions in ReLU neural networks is another field of application [27].…”

Section: Related Workmentioning

confidence: 99%

Advances in verification of ReLU neural networks

Rössig

Petković

2020

J Glob Optim

View full text Add to dashboard Cite

We consider the problem of verifying linear properties of neural networks. Despite their success in many classification and prediction tasks, neural networks may return unexpected results for certain inputs. This is highly problematic with respect to the application of neural networks for safety-critical tasks, e.g. in autonomous driving. We provide an overview of algorithmic approaches that aim to provide formal guarantees on the behaviour of neural networks. Moreover, we present new theoretical results with respect to the approximation of ReLU neural networks. On the other hand, we implement a solver for verification of ReLU neural networks which combines mixed integer programming with specialized branching and approximation techniques. To evaluate its performance, we conduct an extensive computational study. For that we use test instances based on the ACAS Xu system and the MNIST handwritten digit data set. The results indicate that our approach is very competitive with others, i.e. it outperforms the solvers of Bunel et al. (in: Bengio, Wallach, Larochelle, Grauman, Cesa-Bianchi, Garnett (eds) Advances in neural information processing systems (NIPS 2018), 2018) and Reluplex (Katz et al. in: Computer aided verification—29th international conference, CAV 2017, Heidelberg, Germany, July 24–28, 2017, Proceedings, 2017). In comparison to the solvers ReluVal (Wang et al. in: 27th USENIX security symposium (USENIX Security 18), USENIX Association, Baltimore, 2018a) and Neurify (Wang et al. in: 32nd Conference on neural information processing systems (NIPS), Montreal, 2018b), the number of necessary branchings is much smaller. Our solver is publicly available and able to solve the verification problem for instances which do not have independent bounds for each input neuron.

show abstract

Empirical Bounds on Linear Regions of Deep Rectifier Networks

Cited by 32 publications

References 25 publications

Tropical Geometry and Machine Learning

Tropical Geometry and Machine Learning

Modeling Design and Control Problems Involving Neural Network Surrogates

Advances in verification of ReLU neural networks

Contact Info

Product

Resources

About