Achieving robustness in classification using optimal transport with hinge regularization

Abstract

We propose a new framework for robust binary classification, with Deep Neural Networks, based on a hinge regularization of the Kantorovich-Rubinstein dual formulation for the estimation of the Wasserstein distance. The robustness of the approach is guaranteed by the strict Lipschitz constraint on functions required by the optimization problem and direct interpretation of the loss in terms of adversarial robustness. We prove that this classification formulation has a solution, and is still the dual formulation of an optimal transportation problem. We also establish the geometrical properties of this optimal solution. We summarize state-of-the-art methods to enforce Lipschitz constraints on neural networks and we propose new ones for convolutional networks (associated with an open source library for this purpose). The experiments show that the approach provides the expected guarantees in terms of robustness without any significant accuracy drop. The results also suggest that adversarial attacks on the proposed models visibly and meaningfully change the input, and can thus serve as an explanation for the classification.Preprint. Under review.

Robust-by-Design Classification via Unitary-Gradient Neural Networks

“…(Miyato et al 2018) achieved 1-Lipschitz fully connected layers by bounding the spectralnorm of the weight matrices to be 1. Similarly, (Serrurier et al 2021) considered neural networks f in which each component f i is 1-Lipschitz, thus, differently from the 1-Lipschitz networks mentioned before, given a sample x, the lower bound of MAP is deduced by 1 2 (f l (x) − f s (x)). Other authors leveraged orthogonal weight matrices to pursue the same objective.…”

“…The Thirty-Seventh AAAI Conference on Artificial Intelligence network output (Tsuzuku, Sato, and Sugiyama 2018). These particular models can be obtained by composing orthogonal layers (Cisse et al 2017;Li et al 2019;Trockman and Kolter 2021;Serrurier et al 2021; and normpreserving activation functions, such as those presented by (Anil, Lucas, and Grosse 2019;Chernodub and Nowicki 2017). However, despite the satisfaction of the Lipschitz inequality, these models do not provide the exact boundary distance but only a lower bound.…”

WASCO: A Wasserstein-based statistical tool to compare conformational ensembles of intrinsically disordered proteins

“…For an introduction to this Theory, we refer to [16]. Most of the applications of OT are related to the very active field of machine learning, notably in the framework of generative networks [17], robustness [18] or fairness [19], among others. With some notable exemptions [5,[20][21][22][23], Wasserstein distance has not been widely used in structural biology.…”

Inversion of Integral Models: a Neural Network Approach

“…A fine characterization of the convergence conditions of recurrent neural network and of their stability via the estimation of a Lipschitz constant is done in [37,38]. In particular, the Lipschitz constant estimated in [38] is more accurate than in basic approaches which often rely in computing the product of the norms of the linear weight operators of each layer as in [39,40]. Thanks to the aforementioned works, proofs of convergence and stability have been demonstrated on specific neural networks applied to inverse problems as in [24,25,34].…”