Image and Model Transformation with Secret Key for Vision Transformer

Abstract

In this paper, we propose a combined use of transformed images and vision transformer (ViT) models transformed with a secret key. We show for the first time that models trained with plain images can be directly transformed to models trained with encrypted images on the basis of the ViT architecture, and the performance of the transformed models is the same as models trained with plain images when using test images encrypted with the key. In addition, the proposed scheme does not require any specially prepared data for training models or network modification, so it also allows us to easily update the secret key. In an experiment, the effectiveness of the proposed scheme is evaluated in terms of performance degradation and model protection performance in an image classification task on the CIFAR-10 dataset.

Effects of JPEG Compression on Vision Transformer Image Classification for Encryption-then-Compression Images

“…Another approach for protecting copyright and privacy information in test images is to conceal the visual information. Image encryption is a typical technique for concealing visual information, and image-encryption methods have been actively studied to train encrypted images using deep neural networks [ 3 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ]. The method in [ 3 ] combines federated learning with image encryption for test images.…”

“…Encrypted image classification via a cloud server assumes that a user encrypts test images and transmits the encrypted images to a server. Thus, it is desirable to be able to compress the encrypted images in terms of the transmission efficiency; however, most such methods [ 3 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] do not consider image compression. Aprilpyone et al employed the encryption-then-compression (EtC) system [ 18 ] as an image encryption algorithm so that the encrypted images (hereafter, EtC images) possess a high compression performance [ 8 ].…”

“…They are known to provide a higher classification accuracy compared with convolutional neural networks, which are the conventional mainstream image-classification models. Kiya et al focused on the properties of ViT to maintain the classification accuracy for encrypted images [ 16 ]. This method prepares a series of encryption keys (hereafter, key set) and uses it to encrypt not only test images but also a trained ViT model.…”

“…As an extension of the method [ 16 ], we previously introduced an EtC system for the image encryption process [ 17 ]. The EtC system is based on a block-wise transformation, and thus the EtC images can maintain high compression performance.…”

Color-NeuraCrypt: Privacy-Preserving Color-Image Classification Using Extended Random Neural Networks

“…However, data privacy such as personal medical records, may be compromised in that process, because a third party can access the uploaded data illegally, so it is necessary to protect data privacy in cloud environments, and privacy-preserving methods for deep learning have become an urgent challange [2]. One of the most efficient solutions is to encrypt data before the data uploading, so that data owners can train and then test their DNNs in a cloud server with the encrypted data directly [3][4][5][6].…”