Large Norms of CNN Layers Do Not Hurt Adversarial Robustness

Abstract

Since the Lipschitz properties of convolutional neural network (CNN) are widely considered to be related to adversarial robustness, we theoretically characterize the 1 norm and ∞ norm of 2D multi-channel convolutional layers and provide efficient methods to compute the exact 1 norm and ∞ norm. Based on our theorem, we propose a novel regularization method termed norm decay, which can effectively reduce the norms of CNN layers. Experiments show that normregularization methods, including norm decay, weight decay, and singular value clipping, can improve generalization of CNNs. However, we are surprised to find that they can slightly hurt adversarial robustness. Furthermore, we compute the norms of layers in the CNNs trained with three different adversarial training frameworks and find that adversarially robust CNNs have comparable or even larger norms than their nonadversarially robust counterparts. Moreover, we prove that under a mild assumption, adversarially robust classifiers can be achieved with neural networks and an adversarially robust neural network can have arbitrarily large Lipschitz constant. For these reasons, enforcing small norms of CNN layers may be neither effective nor necessary in achieving adversarial robustness. Our code is available at https://github.com/ youweiliang/norm_robustness.

On the Properties of Adversarially-Trained CNNs

“…The method is based on the optimization of a loss taking into account both natural accuracy and the adversarial robustness. Finally, another line of research is devoted to the analysis and estimation of the Lipschitz constant of DNNs, as a guarantee of stability and robustness to be enforced during training [23,24,25]. The present work complements prior studies on the properties of robust models and provide arguments to resolve contrasting results found in the literature.…”

“…Our findings suggest a novel direction for investigation about the mechanism through which local robustness may be implemented by adversarially-trained CNNs, namely a coupling between feature maps. Notice that in [25], the authors propose -altough for a very simple DNNa coupling between subsequent layers as a viable solution for achieving small Lipschitz constants, regardless of their norm. Results presented above suggest that AT might exploit similar schemes.…”