We study the modeling of non-convex piecewise linear functions as Mixed Integer Programming (MIP) problems. We review several new and existing MIP formulations for continuous piecewise linear functions with special attention paid to multivariate non-separable functions. We compare these formulations with respect to their theoretical properties and their relative computational performance. In addition, we study the extension of these formulations to lower semicontinuous piecewise linear functions.

x2 x1 y x2 x1 y Fig. 1: The convex relaxation for a ReLU neuron using: (Left) existing MIP formulations, and (Right) the formulations presented in this paper.Abstract. We present an ideal mixed-integer programming (MIP) formulation for a rectified linear unit (ReLU) appearing in a trained neural network. Our formulation requires a single binary variable and no additional continuous variables beyond the input and output variables of the ReLU. We contrast it with an ideal "extended" formulation with a linear number of additional continuous variables, derived through standard techniques. An apparent drawback of our formulation is that it requires an exponential number of inequality constraints, but we provide a routine to separate the inequalities in linear time. We also prove that these exponentially-many constraints are facet-defining under mild conditions. Finally, we study network verification problems and observe that dynamically separating from the exponential inequalities 1) is much more computationally efficient and scalable than the extended formulation, 2) decreases the solve time of a state-of-the-art MIP solver by a factor of 7 on smaller instances, and 3) nearly matches the dual bounds of a state-of-the-art MIP solver on harder instances, after just a few rounds of separation and in orders of magnitude less time.

No abstract

1. Introduction. Throughout more than 50 years of existence, mixed integer linear programming (MIP) theory and practice have been significantly developed, and it is now an indispensable tool in business and engineering [68,94,104]. Two reasons for the success of MIP are linear programming (LP) based solvers and the modeling flexibility of MIP. We now have several extremely effective state-of-the-art solvers [82,69, 52,171] that incorporate many advanced techniques [1,2,25,23,92,112,24] and, since its early stages, MIP has been used to model a wide range of applications [44,45].While in many cases constructing valid MIP formulations is relatively straightforward, some care should be taken in this construction as certain formulation attributes can significantly reduce the effectiveness of LP-based solvers. Fortunately, constructing formulations that behave well with state-of-the-art solvers can usually be achieved by following simple guidelines described in standard textbooks. However, more advanced techniques can often perform significantly better than textbook formulations and are sometimes a necessity. The main objective of this survey is to summarize the state of the art of such formulation techniques for a wide range of problems. To keep the length of this survey under control, we concentrate on formulations for sets of a mixed integer nature that require both integer constrained and continuous variables. We hence purposefully place less emphasis on some related areas such as combinatorial optimization, quadratic and polynomial 0/1 optimization, and polyhedral approximations of convex sets. These topics are certainly areas of important and active research, so we cover them succinctly in section 12.