Strong mixed-integer programming formulations for trained neural networks

Anderson, Ross; Huchette, Joey; Ma, Will; Tjandraatmadja, Christian; Vielma, Juan Pablo

doi:10.48550/arxiv.1811.01988

Cited by 5 publications

(11 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Any method that improves on any of the above issues can possibly bypass the barrier; see, e.g., SDPbased verifiers [Raghunathan et al, 2018b] can consider the interaction between each neuron within one layer; [Anderson et al, 2018] can relax the combination of one ReLU layer and one affine layer.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

“…We emphasize that by optimal, we mean the optimal convex relaxation of the single nonlinear constraint x (l+1) = σ (l) (z (l) ) (see Proposition (B.3)) instead of the optimal convex relaxation of the nonconvex feasible set of the original problem (O). As such, techniques as in [Anderson et al, 2018, Raghunathan et al, 2018b] are outside our framework; see appendix C for more discussions.…”

Section: Convex Relaxation From the Primal Viewmentioning

confidence: 99%

“…In fact, for neural networks with more than two hidden layers (L ≥ 2), the optimal convex relaxation of the nonconvex feasible set of problem (O) is a strict subset of the feasible set of problem (C), even with the tightest bounds (z [L] , z [L] ) and the optimal choice of σ (l) opt (z) and σ (l) opt (z) in ( 6). It is possible to obtain other (maybe tighter) convex relaxations [Anderson et al, 2018], but it comes with more assumptions on the nonlinear layers and more complex convex constraints.…”

Section: Convex Relaxations Not Included In Problem (C)mentioning

confidence: 99%

See 2 more Smart Citations

A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks

Salman,

Yang,

Zhang

et al. 2019

Preprint

View full text Add to dashboard Cite

Verification of neural networks enables us to gauge their robustness against adversarial attacks. Verification algorithms fall into two categories: exact verifiers that run in exponential time and relaxed verifiers that are efficient but incomplete. In this paper, we unify all existing LP-relaxed verifiers, to the best of our knowledge, under a general convex relaxation framework. This framework works for neural networks with diverse architectures and nonlinearities and covers both primal and dual views of neural network verification. Next, we perform large-scale experiments, amounting to more than 22 CPU-years, to obtain exact solution to the convex-relaxed problem that is optimal within our framework for ReLU networks. We find the exact solution does not significantly improve upon the gap between PGD and existing relaxed verifiers for various networks trained normally or robustly on MNIST and CIFAR datasets. Our results suggest there is an inherent barrier to tight verification for the large class of methods captured by our framework. We discuss possible causes of this barrier and potential future directions for bypassing it. Our code and trained models are available at http://github.com/Hadisalman/robust-verify-benchmark.

show abstract

Section: Conclusion and Discussionmentioning

confidence: 99%

Section: Convex Relaxation From the Primal Viewmentioning

confidence: 99%

Section: Convex Relaxations Not Included In Problem (C)mentioning

confidence: 99%

See 1 more Smart Citation

A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks

Salman,

Yang,

Zhang

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…• Strong MILP formulations and a related family of cutting planes for ReLU networks were recently presented in [56]. These advancements could extend the applicability of ReLU networks as surrogate models by lowering solution times.…”

Section: Promising Research Directionsmentioning

confidence: 99%

ReLU networks as surrogate models in mixed-integer linear programs

Grimstad

Andersson

2019

Computers & Chemical Engineering

109

View full text Add to dashboard Cite

We consider the embedding of piecewise-linear deep neural networks (ReLU networks) as surrogate models in mixed-integer linear programming (MILP) problems. A MILP formulation of ReLU networks has recently been applied by many authors to probe for various model properties subject to input bounds. The formulation is obtained by programming each ReLU operator with a binary variable and applying the big-M method. The efficiency of the formulation hinges on the tightness of the bounds defined by the big-M values. When ReLU networks are embedded in a larger optimization problem, the presence of output bounds can be exploited in bound tightening. To this end, we devise and study several bound tightening procedures that consider both input and output bounds. Our numerical results show that bound tightening may reduce solution times considerably, and that small-sized ReLU networks are suitable as surrogate models in mixed-integer linear programs.

show abstract

“…There are, however, proposed uses of Mixed-Integer and Linear Programming technology in other aspects of Deep Learning. Some examples of this include feature visualization [Fischetti and Jo, 2018], generating adversarial examples [Cheng et al, 2017, Khalil et al, 2018, Fischetti and Jo, 2018, counting linear regions of a Deep Neural Network [Serra et al, 2017], performing inference [Amos et al, 2017] and providing strong convex relaxations for trained neural networks [Anderson et al, 2018].…”

Section: Related Workmentioning

confidence: 99%

Principled Deep Neural Network Training through Linear Programming

Bienstock¹,

Muñoz²,

Pokutta³

2018

Preprint

View full text Add to dashboard Cite

Deep Learning has received significant attention due to its impressive performance in many state-of-theart learning tasks. Unfortunately, while very powerful, Deep Learning is not well understood theoretically and in particular only recently results for the complexity of training deep neural networks have been obtained. In this work we show that large classes of deep neural networks with various architectures (e.g., DNNs, CNNs, Binary Neural Networks, and ResNets), activation functions (e.g., ReLUs and leaky ReLUs), and loss functions (e.g., Hinge loss, Euclidean loss, etc) can be trained to near optimality with desired target accuracy using linear programming in time that is exponential in the input data and parameter space dimension and polynomial in the size of the data set; improvements of the dependence in the input dimension are known to be unlikely assuming P N P, and improving the dependence on the parameter space dimension remains open. In particular, we obtain polynomial time algorithms for training for a given fixed network architecture. Our work applies more broadly to empirical risk minimization problems which allows us to generalize various previous results and obtain new complexity results for previously unstudied architectures in the proper learning setting.

show abstract

Strong mixed-integer programming formulations for trained neural networks

Cited by 5 publications

References 39 publications

A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks

A Convex Relaxation Barrier to Tight Robustness Verification of Neural Networks

ReLU networks as surrogate models in mixed-integer linear programs

Principled Deep Neural Network Training through Linear Programming

Contact Info

Product

Resources

About