Revealing the Structure of Deep Neural Networks via Convex Duality

Ergen, Tolga; Pilancı, Mert

doi:10.48550/arxiv.2002.09773

Cited by 3 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since the publication of Papyan, Han, and Donoho [2020], several works [Mixon et al, 2020, Lu and Steinerberger, 2020, E and Wojtowytsch, 2020, Poggio and Liao, 2020a,b, Fang et al, 2021, Ergen and Pilanci, 2020, Zhu et al, 2021 proposed mathematical derivations and frameworks ratifying the empirical facts of Neural Collapse. In Section E of the Supplementary Material, we survey each of these very recent papers-which are currently only available in preprint.…”

Section: Related Nc Workmentioning

confidence: 99%

“…Thus, they might ultimately appear with very different claims or results. Additionally, works such as Poggio and Liao [2020a,b], Ergen and Pilanci [2020] also analyze behaviors other than NC; we will only discuss the parts relevant to Neural Collapse here.…”

Section: E Related Work Examining Neural Collapsementioning

confidence: 99%

“…While the above-described works tend to focus on either the used-in-practice cross-entropy loss or the theoretically-insightful MSE loss, Ergen and Pilanci [2020] observed that these are both instances of the general class of convex loss functions and, thus, one could derive insights from the classical convex analysis literature. Moreover, compared to Mixon et al [2020], Lu and Steinerberger [2020], E and Wojtowytsch [2020], this work studies the optimization starting from the second-to-last layer features rather than the last-layer features.…”

Section: E5 Ergen and Pilanci [2020]mentioning

confidence: 99%

“…Since strong-duality only characterizes properties of the converged optimal solution of an optimization model, Ergen and Pilanci [2020] does not provide insights into the dynamics with which that solution is achieved which training.…”

Section: E5 Ergen and Pilanci [2020]mentioning

confidence: 99%

“…The motivating philosophy is that, after raw inputs are passed through some initial number of layers, the overparameterization of those layers would allow us to effectively model the N -th-to-last layer features as freely-moving in some subset of Euclidean space. In this terminology, the concurrent works of Mixon et al [2020], Lu and Steinerberger [2020], E and Wojtowytsch [2020] on the unconstrained (last-layer) features model could be considered instances of a 1-layer-peeled model; while the model of Ergen and Pilanci [2020] could be considered as a 2-layer-peeled model. This perspective is attractive because it gives a name and organization to a common modeling philosophy behind the above-described body of independent works.…”

Section: E5 Ergen and Pilanci [2020]mentioning

confidence: 99%

See 4 more Smart Citations

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

Han¹,

Papyan²,

Donoho³

2021

Preprint

View full text Add to dashboard Cite

Recent work [Papyan, Han, and Donoho, 2020] discovered a phenomenon called Neural Collapse (NC) that occurs pervasively in today's deep net training paradigm of driving cross-entropy loss towards zero. In this phenomenon, the last-layer features collapse to their class-means, both the classifiers and class-means collapse to the same Simplex Equiangular Tight Frame (ETF), and the behavior of the last-layer classifier converges to that of the nearest-class-mean decision rule. Since then, follow-ups-such as Mixon et al. [2020] and Poggio and Liao [2020a,b]formally analyzed this inductive bias by replacing the hard-to-study cross-entropy by the more analytically tractable mean squared error (MSE) loss. But, these works stopped short of demonstrating the empirical reality of MSE-NC on benchmark datasets and canonical networks-as had been done in Papyan, Han, and Donoho [2020] for the cross-entropy loss. In this work, we establish the empirical reality and reproducibility of MSE-NC by reporting experimental observations for three prototypical networks and five canonical datasets-with code for reproducing NC in PyTorch for both MSE and cross-entropy in the following Google Colaboratory notebook: here. Following this, we develop three main contributions inspired by MSE-NC. Firstly, we show a new theoretical decomposition of the MSE loss into (A) a term assuming the last-layer classifier is exactly the least-squares or Webb and Lowe [1990] classifier and (B) a term capturing the deviation from this least-squares classifier. Secondly, we exhibit experiments on canonical datasets and networks demonstrating that, during training, term-(B) is negligible. This motivates a new theoretical construct: the central path, where the linear classifier stays MSE-optimal-for the given feature activations-throughout the dynamics. Finally, through our study of continually renormalized gradient flow along the central path, we produce closed-form dynamics that predict full Neural Collapse in an unconstrained features model.

show abstract

Section: Related Nc Workmentioning

confidence: 99%

Section: E Related Work Examining Neural Collapsementioning

confidence: 99%

Section: E5 Ergen and Pilanci [2020]mentioning

confidence: 99%

Section: E5 Ergen and Pilanci [2020]mentioning

confidence: 99%

Section: E5 Ergen and Pilanci [2020]mentioning

confidence: 99%

See 3 more Smart Citations

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

Han¹,

Papyan²,

Donoho³

2021

Preprint

View full text Add to dashboard Cite

show abstract

Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time

Bartan¹,

Pilancı²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

The training of two-layer neural networks with nonlinear activation functions is an important non-convex optimization problem with numerous applications and promising performance in layerwise deep learning. In this paper, we develop exact convex optimization formulations for two-layer neural networks with second degree polynomial activations based on semidefinite programming. Remarkably, we show that semidefinite lifting is always exact and therefore computational complexity for global optimization is polynomial in the input dimension and sample size for all input data. The developed convex formulations are proven to achieve the same global optimal solution set as their non-convex counterparts. More specifically, the globally optimal two-layer neural network with polynomial activations can be found by solving a semidefinite program (SDP) and decomposing the solution using a procedure we call Neural Decomposition. Moreover, the choice of regularizers plays a crucial role in the computational tractability of neural network training. We show that the standard weight decay regularization formulation is NP-hard, whereas other simple convex penalties render the problem tractable in polynomial time via convex programming. We extend the results beyond the fully connected architecture to different neural network architectures including networks with vector outputs and convolutional architectures with pooling. We provide extensive numerical simulations showing that the standard backpropagation approach often fails to achieve the global optimum of the training loss. The proposed approach is significantly faster to obtain better test accuracy compared to the standard backpropagation procedure.

show abstract

Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

Bartan,

Pilanci

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware. Although post-training quantization is well-studied, training optimal quantized NNs involves combinatorial non-convex optimization problems which appear intractable. In this work, we introduce a convex optimization strategy to train quantized NNs with polynomial activations. Our method leverages hidden convexity in twolayer neural networks from the recent literature, semidefinite lifting, and Grothendieck's identity. Surprisingly, we show that certain quantized NN problems can be solved to global optimality in polynomial-time in all relevant parameters via semidefinite relaxations. We present numerical examples to illustrate the effectiveness of our method. of the SDP. Our techniques lead to a provable guarantee for the difference between the resulting loss and the optimal non-convex loss for the first time. Prior workRecently, there has been a lot of research effort in the realm of compression and quantization of neural networks for real-time implementations. In [41], the authors proposed a method that reduces network weights into ternary values by performing training with ternary values. Experiments in [41] show that their method does not suffer from performance degradation and achieve 16x compression compared to the original model. The authors in [18] focus on compressing dense layers using quantization to tackle model storage problems for large-scale models. The work presented in [21] also aims to compress deep networks using a combination of pruning, quantization and Huffman coding. In [30], the authors present a quantization scheme where they use different bit-widths for different layers (i.e., bit-width optimization). Other works that deal with fixed point training include [29], [19], [22]. Furthermore, [4] proposes layer-wise quantization based on ℓ 2 -norm error minimization followed by retraining of the quantized weights. However, these studies do not address optimal approximation. In comparison, our approach provides optimal quantized neural networks.In [2], it was shown that the degree two polynomial activation functions perform comparably to ReLU activation in practical deep networks. Specifically, it was reported in [2] that for deep neural networks, ReLU activation achieves a classification accuracy of 0.96 and a degree two polynomial activation yields an accuracy of 0.95 on the Cifar-10 dataset. Similarly for the Cifar-100 dataset, it is possible to obtain an accuracy of 0.81 for ReLU activation and 0.76 for the degree two polynomial activation. These numerical results are obtained for the activation σ(t) = t + 0.1t 2 . Furthermore, in encrypted computing, it is desirable to have low degree polynomials as activation functions. For instance, homomorphic encryption methods can only support additions and multiplications in a straightforward way. These constraints mak...

show abstract

Revealing the Structure of Deep Neural Networks via Convex Duality

Cited by 3 publications

References 0 publications

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time

Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

Contact Info

Product

Resources

About