A New Look at an Old Problem: A Universal Learning Approach to Linear Regression

Bibas, Koby; Fogel, Yaniv; Feder, Meir

doi:10.1109/isit.2019.8849398

Cited by 21 publications

(17 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results, together with other work in the last year [6][7][8][9], provide a significant understanding of the ramifications of selecting interpolating solutions of noisy data generated from a high-dimensional, or overparameterized linear model, i.e. where the number of features used far exceeds the number of samples of training data.…”

Section: Conclusion For High-dimensional Generative Modelmentioning

confidence: 60%

“…An earlier edition of our work was presented at Information Theory and Applications, February 2019 and subsequently accepted to IEEE International Symposium on Information Theory, July 2019. Several elegant and interesting papers [6][7][8][9][10]46] have appeared around this time. All of these center around the analysis of the 2 -minimizing interpolator.…”

Section: Concurrent Work In High-dimensional Linear Regressionmentioning

confidence: 99%

“…Bibas, Fogel and Feder [9] study online/universal linear regression in the overparameterized regime and show that when the high-dimensional training data is mostly along certain pre-determined directions that favor the true parameter vector, good generalization is possible with both 2 -regularized and minimum 2 -norm interpolators used for prediction. [10] also studies the asymptotically overparameterized regime and obtains precise analytical risk curves for both the 2 -minimizing interpolator and the 1 -minimizing interpolator, with special focus on the overfitting peak at the "interpolation threshold", i.e.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Harmless Interpolation of Noisy Data in Regression

Muthukumar¹,

Vodrahalli²,

Subramanian³

et al. 2020

IEEE J. Sel. Areas Inf. Theory

View full text Add to dashboard Cite

A continuing mystery in understanding the empirical success of deep neural networks is their ability to achieve zero training error and generalize well, even when the training data is noisy and there are more parameters than data points. We investigate this overparameterized regime in linear regression, where all solutions that minimize training error interpolate the data, including noise. We characterize the fundamental generalization (mean-squared) error of any interpolating solution in the presence of noise, and show that this error decays to zero with the number of features. Thus, overparameterization can be explicitly beneficial in ensuring harmless interpolation of noise. We discuss two root causes for poor generalization that are complementary in nature -signal "bleeding" into a large number of alias features, and overfitting of noise by parsimonious feature selectors. For the sparse linear model with noise, we provide a hybrid interpolating scheme that mitigates both these issues and achieves order-optimal MSE over all possible interpolating solutions. arXiv:1903.09139v2 [cs.LG] 9 Sep 2019 2. We provide a Fourier-theoretic interpretation of concurrent analyses [6-10] of the minimum 2 -norm interpolator.3. We show (Theorem 2) that parsimonious interpolators (like the 1 -minimizing interpolator and its relatives) suffer the complementary problem of overfitting pure noise.4. We construct two-step hybrid interpolators that successfully recover signal and harmlessly fit noise, achieving the order-optimal rate of test MSE among all interpolators (Proposition 1 and all its corollaries). Related workWe discuss prior work in three categories: a) overparameterization in deep neural networks, b) interpolation of high-dimensional data using kernels, and c) high-dimensional linear regression. We then recap work on overparameterized linear regression that is concurrent to ours. Recent interest in overparameterizationConventional statistical wisdom is that using more parameters in one's model than data points leads to poor generalization. This wisdom is corroborated in theory by worst-case generalization bounds on such overparameterized models following from VC-theory in classification [2] and ill-conditioning in least-squares regression [5]. It is, however, contradicted in practice by the notable recent trend of empirically successful overparameterized deep neural networks. For example, the commonly used CIFAR-10 dataset contains 60000 images, but the number of parameters in all the neural networks achieving state-of-the-art performance on CIFAR-10 is at least 1.5 million [4]. These neural networks have the ability to memorize pure noisesomehow, they are still able to generalize well when trained with meaningful data.Since the publication of this observation [4,11], the machine learning community has seen a flurry of activity to attempt to explain this phenomenon, both for classification and regression problems, in neural networks. The problem is challenging for three core reasons 2 :1. The optimization landscape for l...

show abstract

Section: Conclusion For High-dimensional Generative Modelmentioning

confidence: 60%

Section: Concurrent Work In High-dimensional Linear Regressionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Harmless Interpolation of Noisy Data in Regression

Muthukumar¹,

Vodrahalli²,

Subramanian³

et al. 2020

IEEE J. Sel. Areas Inf. Theory

View full text Add to dashboard Cite

show abstract

“…Several methods deal with obtaining the pNML learner for different hypothesis sets. Bibas et al (2019b) and Bibas and Feder (2021) showed the pNML solution for linear regression. Rosas et al (2020) proposed an NML based decision strategy for supervised classification problems and showed it attains heuristic PAC learning.…”

Section: Dnn Adaptationmentioning

confidence: 99%

“…Follow this procedure for every label and normalize to get a valid probability assignment. The pNML was developed before for linear regression (Bibas et al, 2019b) and was evaluated empirically for DNN (Fu and Levine, 2021;Bibas et al, 2019a).…”

Section: Introductionmentioning

confidence: 99%

Single Layer Predictive Normalized Maximum Likelihood for Out-of-Distribution Detection

Bibas¹,

Feder²,

Hassner³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Detecting out-of-distribution (OOD) samples is vital for developing machine learning based models for critical safety systems. Common approaches for OOD detection assume access to some OOD samples during training which may not be available in a real-life scenario. Instead, we utilize the predictive normalized maximum likelihood (pNML) learner, in which no assumptions are made on the tested input. We derive an explicit expression of the pNML and its generalization error, denoted as the regret, for a single layer neural network (NN). We show that this learner generalizes well when (i) the test vector resides in a subspace spanned by the eigenvectors associated with the large eigenvalues of the empirical correlation matrix of the training data, or (ii) the test sample is far from the decision boundary. Furthermore, we describe how to efficiently apply the derived pNML regret to any pretrained deep NN, by employing the explicit pNML for the last layer, followed by the softmax function. Applying the derived regret to deep NN requires neither additional tunable parameters nor extra data. We extensively evaluate our approach on 74 OOD detection benchmarks using DenseNet-100, ResNet-34, and WideResNet-40 models trained with CIFAR-100, CIFAR-10, SVHN, and ImageNet-30 showing a significant improvement of up to 15.6% over recent leading methods.

show abstract