Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networks

Training Robust Neural Networks Using Lipschitz Bounds

“…and the second inequality, i.e., (13), is implied by (15). This shows that ( 15) implies (12c) for unpadded and same-padded signals.…”

“…We denote the inequality (15) by G k (X k−1 , X k , ν k ) ⪰ 0, where ν k = P k contains the slack variables in (15). We further note that Lemma 2 is lossless in the cases d = 0, 1, i. e., (13) holds for the k-th layer if and only if (15) is satisfied.…”

“…We denote the inequality (15) by G k (X k−1 , X k , ν k ) ⪰ 0, where ν k = P k contains the slack variables in (15). We further note that Lemma 2 is lossless in the cases d = 0, 1, i. e., (13) holds for the k-th layer if and only if (15) is satisfied. For d ≥ 2, this is no longer the case and the conservatism of (15) might depend on the choice of the realization [17] and the blockdiagonal structure of P k .…”

“…The proof of Lemma 2 is independent of the structure of A k , B k , C k , and D k , which mark the difference between 1-D convolutional and state space model layers. Accordingly, we use (15)…”

“…However, their scalability to deep state-of-the-art NNs is an open research problem which is actively investigated. [12], [13] develop more scalable SDP solvers and [14] do so specific to the problem of SDP-based Lipschitz constant estimation, [15] exploit the chordal sparsity pattern of the underlying linear matrix inequality (LMI) constraint for fully connected NNs and [16], [17] exploit the structure of convolutions. This paper builds on [10], [16], [17] to develop a general, accurate, and scalable SDP-based method for Lipschitz constant estimation for a large family of NN architectures.…”