Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning

Abstract

Contrastive learning (CL) has recently emerged as an effective approach to learning representation in a range of downstream tasks. Central to this approach is the selection of positive (similar) and negative (dissimilar) sets to provide the model the opportunity to 'contrast' between data and class representation in the latent space. In this paper, we investigate CL for improving model robustness using adversarial samples. We first designed and performed a comprehensive study to understand how adversarial vulnerability behaves in the latent space. Based on these empirical evidences, we propose an effective and efficient supervised contrastive learning to achieve model robustness against adversarial attacks. Moreover, we propose a new sample selection strategy that optimizes the positive/negative sets by removing redundancy and improving correlation with the anchor. Experiments conducted on benchmark datasets show that our Adversarial Supervised Contrastive Learning (ASCL) approach outperforms the state-of-the-art defenses by 2.6% in terms of the robust accuracy, whilst our ASCL with the proposed selection strategy can further gain 1.4% improvement with only 42.8% positives and 6.3% negatives compared with ASCL without a selection strategy. 1

Adversarial Contrastive Learning by Permuting Cluster Assignments

“…A number of works utilize adversarially constructed samples in contrastive learning to increase the robustness of the learned feature representations Bui et al [2021], Kim et al [2020], Jiang et al [2020], Chen et al [2020b], McDermott et al [2021], Chen et al [2020a]. RoCL Kim et al [2020] introduces small perturbations to generate adversarial samples, leading to a more robust contrastive model.…”

“…Chen et al [2020b] analyze robustness gains after applying adversarial training in both self-supervised pre-training and supervised fine-tuning, and find that adversarial pre-training contributes the most in robustness improvements. Bui et al [2021] propose a minimax local-global algorithm to generate adversarial samples that improves the performance of SimCLR Chen et al [2020c]. To the best of our knowledge, none of the previous works considered designing more targeted contrastive adversarial examples by leveraging useful cluster structures observed in the data.…”

“…Nevertheless, most instance-wise and clusterbased methods presented lack robustness to adversarial examples. A few works target improving robustness in contrastive learning by devising adversarial perturbations of positive and negative examples Kim et al [2020], Jiang et al [2020], McDermott et al [2021], Bui et al [2021]. These efforts try to design adversarially robust self-supervised learning frameworks by generating perturbations that maximize the contrastive loss of augmented instance-level samples, causing the model to make incorrect predictions.…”

ARIA: Adversarially Robust Image Attribution for Content Provenance

“…Several popular self-supervised learning approaches are based on contrastive learning: Sim-CLR [13], MoCo [28], and BYOL [26]. Most robust selfsupervised approaches focus on robust transfer learning [12,24,31,34,36,38,64] or multi-objective optimization [8,32,44] to improve adversarial robustness. The focus of these works differ from our focus on image retrieval.…”