You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle

Abstract

Deep learning achieves state-of-the-art results in many tasks in computer vision and natural language processing. However, recent works have shown that deep networks can be vulnerable to adversarial perturbations, which raised a serious robustness issue of deep networks. Adversarial training, typically formulated as a robust optimization problem, is an effective way of improving the robustness of deep networks. A major drawback of existing adversarial training algorithms is the computational overhead of the generation of adversarial examples, typically far greater than that of the network training. This leads to the unbearable overall computational cost of adversarial training. In this paper, we show that adversarial training can be cast as a discrete time differential game. Through analyzing the Pontryagin's Maximum Principle (PMP) of the problem, we observe that the adversary update is only coupled with the parameters of the first layer of the network. This inspires us to restrict most of the forward and back propagation within the first layer of the network during adversary updates. This effectively reduces the total number of full forward and backward propagation to only one for each group of adversary updates. Therefore, we refer to this algorithm YOPO (You Only Propagate Once). Numerical experiments demonstrate that YOPO can achieve comparable defense accuracy with approximately 1/5 ∼ 1/4 GPU time of the projected gradient descent (PGD) algorithm [15] . 2 * Equal Contribution 2 Our codes are available at https://github.com/a1600012888/YOPO-You-Only-Propagate-Once Preprint. Under review.

Robust Unpaired Single Image Super-Resolution of Faces

“…Due to SISR being a regression task, we can only compare our adversarial attack with FGSM (Goodfellow, Shlens, and Szegedy 2015) and PGD (Madry et al 2018), since the more recent alternatives, such as TRADES (Zhang et al 2019b) and YOPO (Zhang et al 2019a), are tailored for classification tasks.…”

The art of defense: letting networks fool the attacker

“…Existing adversarial defense methods can be roughly divided into two classes: attacking stage defense and testing stage defense. The adversarial training [41,62,51,28] is an effective way to improve model's robustness, which have two stages defense effect. There are other defense methods have two stages defense effect.…”

Tightening the Approximation Error of Adversarial Risk with Auto Loss Function Search

“…Madry et al [10] developed it by training deep models with stronger adversaries generated by PGD. Subsequent works mainly focus on accelerating training [27], [28] and improving the resistance [29], [30], [31]. Zhang et al [32] characterized the trade-off between accuracy and robustness and proposed TRADES, which optimizes a regularized surrogate loss to improve the adversarial robustness of DNNs.…”