Strong mixed-integer programming formulations for trained neural networks

Anderson, Ross; Huchette, Joey; Ma, Will; Tjandraatmadja, Christian; Vielma, Juan Pablo

doi:10.1007/s10107-020-01474-5

Cited by 99 publications

(34 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since PEREGRiNN is a sound and complete verification algorithm, we restrict our comparison to other sound and complete algorithms. NN verifiers can be grouped into roughly three categories: (i) SMT-based methods, which encode the problem into a Satisfiability Modulo Theory problem [11,18,19] problem as a Mixed Integer Linear Program [3,[5][6][7][8]14,23,29]; (iii) Reachability based methods, which perform layer-by-layer reachability analysis to compute the reachable set [4,13,15,17,30,32,34,35]; and (iv) convex relaxations methods [10,31,33]. In general, (i), (ii) and (iii) suffer from poor scalability.…”

Section: Related Workmentioning

confidence: 99%

PEREGRiNN: Penalized-Relaxation Greedy Neural Network Verifier

Khedr

Ferlez

Shoukry

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Neural Networks (NNs) have increasingly apparent safety implications commensurate with their proliferation in real-world applications: both unanticipated as well as adversarial misclassifications can result in fatal outcomes. As a consequence, techniques of formal verification have been recognized as crucial to the design and deployment of safe NNs. In this paper, we introduce a new approach to formally verify the most commonly considered safety specifications for ReLU NNs – i.e. polytopic specifications on the input and output of the network. Like some other approaches, ours uses a relaxed convex program to mitigate the combinatorial complexity of the problem. However, unique in our approach is the way we use a convex solver not only as a linear feasibility checker, but also as a means of penalizing the amount of relaxation allowed in solutions. In particular, we encode each ReLU by means of the usual linear constraints, and combine this with a convex objective function that penalizes the discrepancy between the output of each neuron and its relaxation. This convex function is further structured to force the largest relaxations to appear closest to the input layer; this provides the further benefit that the most “problematic” neurons are conditioned as early as possible, when conditioning layer by layer. This paradigm can be leveraged to create a verification algorithm that is not only faster in general than competing approaches, but is also able to verify considerably more safety properties; we evaluated PEREGRiNN on a standard MNIST robustness verification suite to substantiate these claims.

show abstract

Section: Related Workmentioning

confidence: 99%

PEREGRiNN: Penalized-Relaxation Greedy Neural Network Verifier

Khedr

Ferlez

Shoukry

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…More complex ML models have also been shown to be MIO-representable, although more effort is required to represent them than simple regression models. Neural networks which use the ReLU activation function can be represented using binary variables and big-M formulations (Amos et al 2016, Grimstad and Andersson 2019, Anderson et al 2020, Chen et al 2020, Spyros 2020, Venzke et al 2020. Where other activation functions are used (Gutierrez-Martinez et al 2011, Lombardi et al 2017, Schweidtmann and Mitsos 2019, the MIO representation of neural networks is still possible, provided the solvers are capable of handling these functions.…”

Section: Literature Reviewmentioning

confidence: 99%

Mixed-Integer Optimization with Constraint Learning

Maragno¹,

Wiberg²,

Bertsimas³

et al. 2021

Preprint

View full text Add to dashboard Cite

We establish a broad methodological foundation for mixed-integer optimization with learned constraints.We propose an end-to-end pipeline for data-driven decision making in which constraints and objectives are directly learned from data using machine learning, and the trained models are embedded in an optimization formulation. We exploit the mixed-integer optimization-representability of many machine learning methods, including linear models, decision trees, ensembles, and multi-layer perceptrons. The consideration of multiple methods allows us to capture various underlying relationships between decisions, contextual variables, and outcomes. We also characterize a decision trust region using the convex hull of the observations, to ensure credible recommendations and avoid extrapolation. We efficiently incorporate this representation using column generation and clustering. In combination with domain-driven constraints and objective terms, the embedded models and trust region define a mixed-integer optimization problem for prescription generation.We implement this framework as a Python package (OptiCL) for practitioners. We demonstrate the method in both chemotherapy optimization and World Food Programme planning. The case studies illustrate the benefit of the framework in generating high-quality prescriptions, the value added by the trust region, the incorporation of multiple machine learning methods, and the inclusion of multiple learned constraints.

show abstract

“…Rectified neural networks are known to be continuous piecewise-linear functions with the universal approximation ability [99]. As a result, they are a subject of interest for capturing complex nonlinear physics in MILPs [51,100]. Neural networks are popular because of their ability to model large high-dimensional data-sets; however they also possess drawbacks when considering their application in modeling for use in an MILP.…”

Section: Combined Unit Commitment and Economic Dispatchmentioning

confidence: 99%

A Critical Review of the Modeling and Optimization of Combined Heat and Power Dispatch

Kazda¹,

Li²

2020

Processes

View full text Add to dashboard Cite

Combined heat and power (CHP) systems are attracting increasing attention for their ability to improve the economics and sustainability of the electricity system. Determining how to best operate these systems is difficult because they can consist of many generating units whose operation is governed by complex nonlinear physics. Mathematical programming is a useful tool to support the operation of CHP systems, and has been the subject of substantial research attention since the early 1990s. This paper critically reviews the modeling and optimization work that has been done on the CHP economic dispatch problem, and the CHP economic and emission dispatch problem. A summary of the common models used for these problems is provided, along with comments on future modeling work that would beneficial to the field. The majority of optimization approaches studied for CHP system operation are metaheuristic algorithms. A discussion of the limitations and benefits of metaheuristic algorithms is given. Finally, a case study optimizing five classic CHP system test instances demonstrates the advantages of the using deterministic global search algorithms over metaheuristic search algorithms.

show abstract

Strong mixed-integer programming formulations for trained neural networks

Cited by 99 publications

References 42 publications

PEREGRiNN: Penalized-Relaxation Greedy Neural Network Verifier

PEREGRiNN: Penalized-Relaxation Greedy Neural Network Verifier

Mixed-Integer Optimization with Constraint Learning

A Critical Review of the Modeling and Optimization of Combined Heat and Power Dispatch

Contact Info

Product

Resources

About