LQF: Linear Quadratic Fine-Tuning

Abstract

Classifiers that are linear in their parameters, and trained by optimizing a convex loss function, have predictable behavior with respect to changes in the training data, initial conditions, and optimization. Such desirable properties are absent in deep neural networks (DNNs), typically trained by non-linear fine-tuning of a pre-trained model. Previous attempts to linearize DNNs have led to interesting theoretical insights, but have not impacted the practice due to the substantial performance gap compared to standard non-linear optimization. We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning on most of real-world image classification tasks tested, thus enjoying the interpretability of linear models without incurring punishing losses in performance. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification: Leaky-ReLU instead of ReLU, mean squared loss instead of cross-entropy, and pre-conditioning using Kronecker factorization. None of these changes in isolation is sufficient to approach the performance of non-linear fine-tuning. When used in combination, they allow us to reach comparable performance, and even superior in the low-data regime, while enjoying the simplicity, robustness and interpretability of linear-quadratic optimization.

Scene Uncertainty and the Wellington Posterior of Deterministic Image Classifiers

“…Instead, we seek for analytical expressions of the Wellington Posterior that do not require multiple inference runs. Recent development in network linearization suggest that it is possible to perturb the weights of a trained network locally to perform novel tasks essentially as well as non-linear optimization/fine-tuning [1]. Such linearization, called LQF, is with respect to perturbations of the weights of the model.…”

“…The viability of the method we propose to impute uncertainty to a deterministic classifier hinges on the ability to produce an estimate without having to sample multiple inference runs at test time. Recent work on model linearization around a pre-trained point [1] has shown that it is possible to obtain performance comparable to that of full-network non-linear finetuning. In this sense, LQF can be used as a baseline classifier instead of the pre-trained network.…”

DIVA: Dataset Derivative of a Learning Task

“…The intermediate block in the diagram (which finds the optimal weights w α for the the training loss on D α ) is usually non-differentiable with respect to the dataset, or the derivative is prohibitively expensive to compute. DIVA leverages recent progress in deep learning linearization [1], to derive a closed-form expression for the derivative of the final loss (validation error) with respect to the dataset weights. In particular, [1] have shown that, by replacing cross-entropy with least-squares, replacing ReLu with leaky-ReLu, and performing suitable pre-conditioning, the linearized model performs on par with full non-linear fine-tuning.…”

“…DIVA leverages recent progress in deep learning linearization [1], to derive a closed-form expression for the derivative of the final loss (validation error) with respect to the dataset weights. In particular, [1] have shown that, by replacing cross-entropy with least-squares, replacing ReLu with leaky-ReLu, and performing suitable pre-conditioning, the linearized model performs on par with full non-linear fine-tuning. We also leverage a classical result to compute the leave-one-out loss of a linear model in closed-form [61,21].…”

“…Rather than using the full linearization of the model derived by [1], we restrict the gradient to its last layer, cognizant that we are not exploiting the full power of LQF and thereby obtaining only a lower-bound of performance improvement. Despite that restriction, our results show consistent improvements from dataset optimization, at the modest computational cost of a forward pass over the dataset to optimize the importance weights.…”