The Convex Relaxation Barrier, Revisited: Tightened Single-Neuron Relaxations for Neural Network Verification

Abstract

We improve the effectiveness of propagation-and linear-optimization-based neural network verification algorithms with a new tightened convex relaxation for ReLU neurons. Unlike previous single-neuron relaxations which focus only on the univariate input space of the ReLU, our method considers the multivariate input space of the affine pre-activation function preceding the ReLU. Using results from submodularity and convex geometry, we derive an explicit description of the tightest possible convex relaxation when this multivariate input is over a box domain. We show that our convex relaxation is significantly stronger than the commonly used univariate-input relaxation which has been proposed as a natural convex relaxation barrier for verification. While our description of the relaxation may require an exponential number of inequalities, we show that they can be separated in linear time and hence can be efficiently incorporated into optimization algorithms on an as-needed basis. Based on this novel relaxation, we design two polynomial-time algorithms for neural network verification: a linearprogramming-based algorithm that leverages the full power of our relaxation, and a fast propagation algorithm that generalizes existing approaches. In both cases, we show that for a modest increase in computational effort, our strengthened relaxation enables us to verify a significantly larger number of instances compared to similar algorithms.Preprint. Under review.

Learning Optimal Prescriptive Trees from Observational Data

“…MIO has been gaining traction as a tool to solve challenging learning problems. It has been used for example to tackle sparse regression problems (Wilson and Sahinidis 2017, Atamtürk and Gómez 2019, Bertsimas et al 2020b, Hazimeh et al 2020, Xie and Deng 2020, Gómez and Prokopyev 2021, verification of neural networks (Fischetti and Jo 2018, Khalil et al 2018, Tjandraatmadja et al 2020, and sparse principal component analysis (Dey et al 2018, Bertsimas et al 2020a), among others. More importantly in the context of this paper, MIO methods have been proposed to learn optimal decision trees (Bertsimas and Dunn 2017, Verwer and Zhang 2019, Aghaei et al 2019, 2020, Elmachtoub et al 2020, Mišić 2020.…”

DeepSplit: Scalable Verification of Deep Neural Networks via Operator Splitting

“…These bounds tend to be loose unless optimized during training, which typically comes at a significant cost to standard performance. Further work has aimed to tighten these bounds [7,31,33], however these works focus primarily on small convolutional networks and struggle to scale to more typical deep networks. Other work has studied the limits of these convex relaxations on these small networks using vast amounts of CPU-compute [28].…”

“…We defer additional analogous results in the smaller MNIST setting to Appendix E.3. CIFAR10 Much work studying verification of deep networks in the CIFAR10 setting [7,31,33], including the more scalable Lagrangian-based methods [2,10], have focused primarily on a CNN with only 6k hidden units from [36]-smaller than the LeNet architecture used for MNIST [21]. Consequently, the LP relaxation for this network can solved exactly with a commercial LP solver such as Gurobi, and recent SDP solvers can produce even tighter bounds [6], rendering any LPbased solutions for this network obsolete.…”

Deep Policy Iteration with Integer Programming for Inventory Management

“…They leverage MP to optimize a mixed-integer (linear) problem (MIP) over a polyhedral action space using commercially available solvers such as CPLEX and Gurobi. A number of papers show how trained ReLU-based DNNs can be expressed as an MP with (Tjandraatmadja et al 2020;Anderson et al 2020) also providing ideal reformulations that improve computational efficiencies with a solver. (Ryu et al 2019) propose a Q-learning framework to optimize over continuous action spaces using a combination of MP and a DNN actor.…”

“…(Ryu et al 2019) propose a Q-learning framework to optimize over continuous action spaces using a combination of MP and a DNN actor. (Delarue, Anderson, and Tjandraatmadja 2020;van Heeswijk and La Poutré 2019;Xu et al 2020) show how to use ReLU-based DNN value functions to optimize combinatorial problems (e.g., vehicle routing) where the immediate rewards are deterministic and the action space is vast. We extend such approaches and results to problems where the immediate reward can be uncertain as is the case with inventory management problems.…”