Offsite-Tuning: Transfer Learning without Full Model

Abstract

Transfer learning is important for foundation models to adapt to downstream tasks. However, many foundation models are proprietary, so users must share their data with model owners to fine-tune the models, which is costly and raise privacy concerns. Moreover, fine-tuning large foundation models is computation-intensive and impractical for most downstream users. In this paper, we propose Offsite-Tuning, a privacy-preserving and efficient transfer learning framework that can adapt billion-parameter foundation models to downstream data without access to the full model. In offsite-tuning, the model owner sends a lightweight adapter and a lossy compressed emulator to the data owner, who then fine-tunes the adapter on the downstream data with the emulator's assistance. The fine-tuned adapter is then returned to the model owner, who plugs it into the full model to create an adapted foundation model. Offsite-tuning preserves both parties' privacy and is computationally more efficient than the existing fine-tuning methods that require access to the full model weights. We demonstrate the effectiveness of offsite-tuning on various large language and vision foundation models. Offsitetuning can achieve comparable accuracy as full model fine-tuning while being privacy-preserving and efficient, achieving 6.5× speedup and 5.6× memory reduction. Code is available here.* https://lambdalabs.com/blog/demystifying-gpt-3

Debiasing Pre-Trained Language Models via Efficient Fine-Tuning

“…We run the same task of a method in the above-mentioned grid search space three times with different random seeds, choose the best result from each run, and report the mean and standard deviation of these best results. For all question-answering tasks, we sweep learning rates in {1, 3, 5, 7}•10 −4 , batch sizes in {8, 16, 32} and the number of epochs in {3, 5, 10}, and keep other settings the same, which is inspired by [48]. The sequence length for all tasks is set to 512, 128, 128 and 128 for BERT base , RoBERTa large , BART large and OPT as our baselines, respectively.…”

DC-CCL: Device-Cloud Collaborative Controlled Learning for Large Vision Models

“…Zaken et al [46] introduced BitFit, a sparse update method that modifies only the bias terms of the model. Xiao et al [42] proposed offsite-tuning, which finetunes the top and bottom layers of model and compresses the large middle layers into a emulator using layer-drop. Lester et al [28] proposed prompt tuning, requiring to store and tune only a small task-specific prompt (i.e., a few tokens) for each downstream task.…”

FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learning

“…Compared to full-parameter fine-tuning, these PEFT algorithms significantly reduce memory consumption, training time, and communication cost for fine-tuning LLMs. Besides, motivated by the practical concerns on intelligent property protection of LLMs, we also integrate a privacy-preserving fine-tuning algorithm, offsite-tuning (Xiao et al, 2023), for the scenario where clients only tune small adapters based on a distilled model from a full LLM.…”

“…To satisfy such practical demand, we adapt a privacy-preserving fine-tuning algorithm, offsitetuning (Xiao et al, 2023), to a federated version, and name it FedOT for short. It sends a lossy compressed model with untrainable parameters to the clients as an emulator of the complete LLM at the beginning of FL.…”

“…In this section, we investigate the performance of federated fine-tuning LLMs without accessing the full model. As mentioned in Section 4.2, we adapt a privacy-preserving fine-tuning algorithm, offsite-tuning (Xiao et al, 2023), to federated scenarios. Specifically, we use the first and last two layers of LLaMA-7B as the adapter and compress the model as the emulator by dropping 20% and 50% of the remaining layers uniformly.…”