Classification vs regression in overparameterized regimes: Does the loss function matter?

Muthukumar, Vidya; Narang, Adhyyan; Subramanian, Vignesh; Belkin, Mikhail; Hsu, Daniel; Sahai, Anant

doi:10.48550/arxiv.2005.08054

Cited by 20 publications

(70 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Overparameterized ML and double-descent The phenomenon of double-descent was first discovered by [6]. This paper and subsequent works on this topic [4,33,32,30,11] emphasize the importance of the right prior (sometimes referred to as inductive bias or regularization) to avail the benefits of overparameterization. However, an important question that arises is: where does this prior come from?…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Towards Sample-efficient Overparameterized Meta-learning

Sun

Narang²,

Gulluk³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

An overarching goal in machine learning is to build a generalizable model with few samples. To this end, overparameterization has been the subject of immense interest to explain the generalization ability of deep nets even when the size of the dataset is smaller than that of the model. While the prior literature focuses on the classical supervised setting, this paper aims to demystify overparameterization for meta-learning. Here we have a sequence of linear-regression tasks and we ask:(1) Given earlier tasks, what is the optimal linear representation of features for a new downstream task? and (2) How many samples do we need to build this representation? This work shows that surprisingly, overparameterization arises as a natural answer to these fundamental meta-learning questions. Specifically, for (1), we first show that learning the optimal representation coincides with the problem of designing a task-aware regularization to promote inductive bias. We leverage this inductive bias to explain how the downstream task actually benefits from overparameterization, in contrast to prior works on few-shot learning. For (2), we develop a theory to explain how feature covariance can implicitly help reduce the sample complexity well below the degrees of freedom and lead to small estimation error. We then integrate these findings to obtain an overall performance guarantee for our meta-learning algorithm. Numerical experiments on real and synthetic data verify our insights on overparameterized meta-learning.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Recent literature in ML theory has posited that overparameterization can be beneficial to generalization in traditional single-task setups for both regression [29,40,4,33,30] and classification [32,31] problems. Empirical literature in deep learning suggests that overparameterization is of interest for both phases of meta-learning as well.…”

Section: Introductionmentioning

confidence: 99%

Towards Sample-efficient Overparameterized Meta-learning

Sun

Narang²,

Gulluk³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We next analyze the classification error of the minimum-Hilbert-norm interpolator of binary labels via a bias-variance decomposition inspired by the work of [17]. We demonstrate an asymptotic separation between the regression and classification tasks akin to that provided in [15]. Although good regression performance implies good classification performance (see Figure 1(b)), the reverse is not true; we characterize regimes in which classification is statistically consistent but regression is not for the case of bounded orthonormal systems.…”

Section: Asymptotic Separation Between Kernel Classification and Regr...mentioning

confidence: 99%

“…In a related line of work, [15], [16] show that the max-margin support-vector-machine can be classification-consistent in the overparametrized setting even when the corresponding regression task does not generalize. These results require even stricter assumptions; in fact, they require independence of the features used in the linear model in the very first step of obtaining sharp expressions for the classification generalization error.…”

Section: Introductionmentioning

confidence: 99%

Harmless interpolation in regression and classification with structured features

McRae¹,

Karnik²,

Davenport³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Overparametrized neural networks tend to perfectly fit noisy training data yet generalize well on test data. Inspired by this empirical observation, recent work has sought to understand this phenomenon of benign overfitting or harmless interpolation in the much simpler linear model. Previous theoretical work critically assumes that either the data features are statistically independent or the input data is high-dimensional; this precludes general nonparametric settings with structured feature maps. In this paper, we present a general and flexible framework for upper bounding regression and classification risk in a reproducing kernel Hilbert space. A key contribution is that our framework describes precise sufficient conditions on the data Gram matrix under which harmless interpolation occurs. Our results recover prior independent-features results (with a much simpler analysis), but they furthermore show that harmless interpolation can occur in more general settings such as features that are a bounded orthonormal system. Furthermore, our results show an asymptotic separation between classification and regression performance in a manner that was previously only shown for Gaussian features.

show abstract

“…Han et al [26] uncovered that the "neural collapse" phenomenon also occurs under square loss where the last-layer features eventually collapse to their simplex-style class-means. Muthukumar et al [27] compared classification and regression tasks in the overparameterized linear model with Gaussian features, illustrating different roles and properties of loss functions used at the training and testing phases. Poggio and Liao [28] made interesting observations on effects of popular regularization techniques such as batch normalization and weight decay on the gradient flow dynamics under square loss.…”

mentioning

confidence: 99%

Understanding Square Loss in Training Overparametrized Neural Network Classifiers

Hu¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning has achieved many breakthroughs in modern classification tasks. Numerous architectures have been proposed for different data structures but when it comes to the loss function, the cross-entropy loss is the predominant choice. Recently, several alternative losses have seen revived interests for deep classifiers. In particular, empirical evidence seems to promote square loss but a theoretical justification is still lacking. In this work, we contribute to the theoretical understanding of square loss in classification by systematically investigating how it performs for overparametrized neural networks in the neural tangent kernel (NTK) regime. Interesting properties regarding the generalization error, robustness, and calibration error are revealed. We consider two cases, according to whether classes are separable or not. In the general non-separable case, fast convergence rate is established for both misclassification rate and calibration error. When classes are separable, the misclassification rate improves to be exponentially fast. Further, the resulting margin is proven to be lower bounded away from zero, providing theoretical guarantees for robustness. We expect our findings to hold beyond the NTK regime and translate to practical settings. To this end, we conduct extensive empirical studies on practical neural networks, demonstrating the effectiveness of square loss in both synthetic low-dimensional data and real image data. Comparing to cross-entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration. introductionThe pursuit of better classifiers has fueled the progress of machine learning and deep learning research. The abundance of benchmark image datasets, e.g., MNIST, CIFAR, ImageNet, etc., provides test fields for all kinds of new classification models, especially those based on deep neural networks (DNN). With the introduction of CNN, ResNets, and transformers, DNN classifiers are constantly improving and catching up to the human-level performance. In contrast to the active innovations in model architecture, the training objective remains largely stagnant, with cross-entropy loss being the default choice. Despite its popularity, cross-entropy has been shown to be problematic in some applications. Among others, Yu et al. [1] argued that features learned from cross-entropy lack interpretability and proposed a new loss aiming for maximum coding rate reduction. Pang et al. [2] linked the use of cross-entropy to adversarial vulnerability and proposed a new classification loss

show abstract

Classification vs regression in overparameterized regimes: Does the loss function matter?

Cited by 20 publications

References 22 publications

Towards Sample-efficient Overparameterized Meta-learning

Towards Sample-efficient Overparameterized Meta-learning

Harmless interpolation in regression and classification with structured features

Understanding Square Loss in Training Overparametrized Neural Network Classifiers

Contact Info

Product

Resources

About