Reexamining Low Rank Matrix Factorization for Trace Norm Regularization

Ciliberto, Carlo; Stamos, Dimitris; Pontil, Massimiliano

doi:10.48550/arxiv.1706.08934

Cited by 4 publications

(7 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to the idea in [35,88,103,127], we will characeterize the critical points of ( 14) by establishing a connection to the optimality condition of the convex problem (37). Towards this goal, we first show the global minimum of the convex program (37) provides a lower bound for the original problem (4).…”

Section: C1 Main Proofmentioning

confidence: 99%

A Geometric Analysis of Neural Collapse with Unconstrained Features

Zhu,

Ding,

Zhou

et al. 2021

Preprint

View full text Add to dashboard Cite

We provide the first global optimization landscape analysis of Neural Collapse -an intriguing empirical phenomenon that arises in the last-layer classifiers and features of neural networks during the terminal phase of training. As recently reported in [1], this phenomenon implies that (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations collapses to zero. We study the problem based on a simplified unconstrained feature model, which isolates the topmost layers from the classifier of the neural network. In this context, we show that the classical cross-entropy loss with weight decay has a benign global landscape, in the sense that the only global minimizers are the Simplex ETFs while all other critical points are strict saddles whose Hessian exhibit negative curvature directions. In contrast to existing landscape analysis for deep neural networks which is often disconnected from practice, our analysis of the simplified model not only does it explain what kind of features are learned in the last layer, but it also shows why they can be efficiently optimized in the simplified settings, matching the empirical observations in practical deep network architectures. These findings could have profound implications for optimization, generalization, and robustness of broad interests. For example, our experiments demonstrate that one may set the feature dimension equal to the number of classes and fix the last-layer classifier to be a Simplex ETF for network training, which reduces memory cost by over 20% on ResNet18 without sacrificing the generalization performance.

show abstract

Section: C1 Main Proofmentioning

confidence: 99%

A Geometric Analysis of Neural Collapse with Unconstrained Features

Zhu,

Ding,

Zhou

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…where 1) is equivalent to trace norm regularization [2], see e.g. [9] and references therein 3 . We follow the formulation of Eq.…”

Section: Methodsmentioning

confidence: 99%

Learning Fair and Transferable Representations

Oneto,

Donini,

Maurer

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Developing learning methods which do not discriminate subgroups in the population is a central goal of algorithmic fairness. One way to reach this goal is by modifying the data representation in order to meet certain fairness constraints. In this work we measure fairness according to demographic parity. This requires the probability of the possible model decisions to be independent of the sensitive information. We argue that the goal of imposing demographic parity can be substantially facilitated within a multitask learning setting. We leverage task similarities by encouraging a shared fair representation across the tasks via low rank matrix factorization. We derive learning bounds establishing that the learned representation transfers well to novel tasks both in terms of prediction performance and fairness metrics. We present experiments on three real world datasets, showing that the proposed method outperforms state-of-the-art approaches by a significant margin.

show abstract

“…We build on this method by adding a variational form of trace norm regularization that was first proposed for collaborative prediction (Srebro et al, 2005) and also applied to recommender systems (Koren et al, 2009). The use of this technique with gradient descent was recently justified theoretically (Ciliberto et al, 2017). Furthermore, Neyshabur et al (2015) argue that trace norm regularization could provide a sensible inductive bias for neural networks.…”

Section: Related Workmentioning

confidence: 99%

“…Unfortunately, there is no known way of directly computing the trace norm and its gradients that would be computationally feasible in the context of large deep learning models. Instead, we propose to combine the two-stage training method of with an indirect variational trace norm regularization technique (Srebro et al, 2005;Ciliberto et al, 2017). We describe this technique in more detail in Section 3.1 and report experimental results in Section 3.2.…”

Section: Training Low Rank Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Trace norm regularization and faster inference for embedded speech recognition RNNs

Kliegl,

Goyal,

Zhao

et al. 2017

Preprint

View full text Add to dashboard Cite

We propose and evaluate new techniques for compressing and speeding up dense matrix multiplications as found in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR). For compression, we introduce and study a trace norm regularization technique for training low rank factored versions of matrix multiplications. Compared to standard low rank training, we show that our method leads to good accuracy versus number of parameter trade-offs and can be used to speed up training of large models. For speedup, we enable faster inference on ARM processors through new open sourced kernels optimized for small batch sizes, resulting in 3x to 7x speed ups over the widely used gemmlowp library. Beyond LVCSR, we expect our techniques and kernels to be more generally applicable to embedded neural networks with large fully connected or recurrent layers.

show abstract

Reexamining Low Rank Matrix Factorization for Trace Norm Regularization

Cited by 4 publications

References 14 publications

A Geometric Analysis of Neural Collapse with Unconstrained Features

A Geometric Analysis of Neural Collapse with Unconstrained Features

Learning Fair and Transferable Representations

Trace norm regularization and faster inference for embedded speech recognition RNNs

Contact Info

Product

Resources

About