Recent years have seen the ever-increasing importance of pre-trained models and their downstream training in deep learning research and applications. At the same time, the defense for adversarial examples has been mainly investigated in the context of training from random initialization on simple classification tasks. To better exploit the potential of pre-trained models in adversarial robustness, this paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks. Existing research has shown that since the robust pre-trained model has already learned a robust feature extractor, the crucial question is how to maintain the robustness in the pre-trained model when learning the downstream task. We study the modelbased and data-based approaches for this goal and find that the two common approaches cannot achieve the objective of improving both generalization and adversarial robustness. Thus, we propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework, which consists of two neural networks where one of them keeps the population means and variances of pre-training data in the batch normalization layers. Besides the robust information transfer, TWINS increases the effective learning rate without hurting the training stability since the relationship between a weight norm and its gradient norm in standard batch normalization layer is broken, resulting in a faster escape from the sub-optimal initialization and alleviating the robust overfitting. Finally, TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness. Our code is available at https://github.com/ziquanliu/ CVPR2023-TWINS. (a) CIFAR10 (b) Caltech-256 Figure 1. The performance of fine-tuning robust and non-robust large-scale pre-trained (PT) ResNet50 [27, 56] on CIFAR10 [35] and Caltech-256 [23]. We compare standard adversarial training (AT), Learning without Forgetting (LwF) (model approach) [40], joint fine-tuning with UOT data selection (data approach) [47]and our TWINS fine-tuning. The robust accuracy is evaluated using l∞ norm bounded AutoAttack [12] with ϵ = 8/255. On CIFAR10, the data-based and model-based approach fail to improve clean and robust accuracy. On Caltech, both approaches improve the clean accuracy but hurt the robust accuracy. Our TWINS fine-tuning improves the clean and robust performance on both datasets. The pink triangle denotes the performance of standard AT with the non-robust pre-trained ResNet50, which drops considerably compared with fine-tuning starting from the robust pretrained model.