LQF: Linear Quadratic Fine-Tuning

Achille, Alessandro; Golatkar, Aditya; Ravichandran, Avinash; Polito, Marzia; Soatto, Stefano

doi:10.1109/cvpr46437.2021.01547

Cited by 8 publications

(8 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where (a) follows from applying Lemma 5 (1) to i (j) t+1 (w t ) and L D (w t ) and (b) follows from applying Lemma 5 (2). Now substituting Equation (25) and Equation (26) in Equation ( 24), we get that,…”

Section: Theoretical Resultsmentioning

confidence: 98%

“…[33] shows that aside from a theoretical tool, it is possible to directly train a (finite) linearized network using an efficient algorithm for the Jacobian-Vector product computation. [2] show that with some changes to the architecture and training process, linearized models can match the performance of non-linear models on many vision tasks, while still maintaining a convex loss function.…”

Section: Related Workmentioning

confidence: 99%

“…In particular, it achieves only slightly better performance than training a linear classifier on top of the last layer features. Following the suggetsion of [2], we replace the ReLUs with Leaky ReLUs, since it boosts the accuracy of linearized models.…”

Section: Optimizing the Mixed-linear Modelmentioning

confidence: 99%

See 2 more Smart Citations

Mixed-Privacy Forgetting in Deep Networks

Golatkar¹,

Achille²,

Ravichandran³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

We show that the influence of a subset of the training samples can be removed -or "forgotten" -from the weights of a network trained on large-scale image classification tasks, and we provide strong computable bounds on the amount of remaining information after forgetting. Inspired by real-world applications of forgetting techniques, we introduce a novel notion of forgetting in mixed-privacy setting, where we know that a "core" subset of the training samples does not need to be forgotten. While this variation of the problem is conceptually simple, we show that working in this setting significantly improves the accuracy and guarantees of forgetting methods applied to vision classification tasks. Moreover, our method allows efficient removal of all information contained in non-core data by simply setting to zero a subset of the weights with minimal loss in performance. We achieve these results by replacing a standard deep network with a suitable linear approximation. With opportune changes to the network architecture and training procedure, we show that such linear approximation achieves comparable performance to the original network and that the forgetting problem becomes quadratic and can be solved efficiently even for large models. Unlike previous forgetting methods on deep networks, ours can achieve close to the state-of-the-art accuracy on large scale vision tasks. In particular, we show that our method allows forgetting without having to trade off the model accuracy.

show abstract

Section: Theoretical Resultsmentioning

confidence: 98%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Mixed-Privacy Forgetting in Deep Networks

Golatkar¹,

Achille²,

Ravichandran³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…tasks (Harutyunyan et al, 2021;Achille et al, 2021). Therefore, studying this setting can provide insights for more general models.…”

Section: Relational Memorization In Linear Modelsmentioning

confidence: 99%

Towards Differential Relational Privacy and its use in Question Answering

Bombari¹,

Achille²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Memorization of the relation between entities in a dataset can lead to privacy issues when using a trained model for question answering. We introduce Relational Memorization (RM) to understand, quantify and control this phenomenon. While bounding general memorization can have detrimental effects on the performance of a trained model, bounding RM does not prevent effective learning. The difference is most pronounced when the data distribution is long-tailed, with many queries having only few training examples: Impeding general memorization prevents effective learning, while impeding only relational memorization still allows learning general properties of the underlying concepts. We formalize the notion of Relational Privacy (RP) and, inspired by Differential Privacy (DP), we provide a possible definition of Differential Relational Privacy (DrP). These notions can be used to describe and compute bounds on the amount of RM in a trained model. We illustrate Relational Privacy concepts in experiments with large-scale models for Question Answering.

show abstract

“…Such public data are distinct from the private ones, for which we seek strong privacy guarantees. 1 In particular, using large amounts of generic public data for pre-training has recently enabled the creation of language models that achieve DP on the target task while remaining close to state-of-the-art performance [34,57]. The same strategy in vision [49], however, still yields punishing error increases (Tab.…”

Section: Introductionmentioning

confidence: 99%

Mixed Differential Privacy in Computer Vision

Golatkar¹,

Achille²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We introduce AdaMix, an adaptive differentially private algorithm for training deep neural network classifiers using both private and public image data. While pre-training language models on large public datasets has enabled strong differential privacy (DP) guarantees with minor loss of accuracy, a similar practice yields punishing trade-offs in vision tasks. A few-shot or even zero-shot learning baseline that ignores private data can outperform fine-tuning on a large private dataset. AdaMix incorporates few-shot training, or cross-modal zero-shot learning, on public data prior to private fine-tuning, to improve the trade-off. AdaMix reduces the error increase from the non-private upper bound from the 167-311% of the baseline, on average across 6 datasets, to 68-92% depending on the desired privacy level selected by the user. AdaMix tackles the trade-off arising in visual classification, whereby the most privacy sensitive data, corresponding to isolated points in representation space, are also critical for high classification accuracy. In addition, AdaMix comes with strong theoretical privacy guarantees and convergence analysis.

show abstract

LQF: Linear Quadratic Fine-Tuning

Cited by 8 publications

References 16 publications

Mixed-Privacy Forgetting in Deep Networks

Mixed-Privacy Forgetting in Deep Networks

Towards Differential Relational Privacy and its use in Question Answering

Mixed Differential Privacy in Computer Vision

Contact Info

Product

Resources

About