Optimal Combination of Linear and Spectral Estimators for Generalized Linear Models

Mondelli, Marco; Thrampoulidis, Christos; Venkataramanan, Ramji

doi:10.48550/arxiv.2008.03326

Cited by 5 publications

(16 citation statements)

References 21 publications

(36 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is in similar flavor to our Theorem 5 which employs a two-stage algorithm. [46,47,53] provide asymptotic/sharp analysis for spectral methods for phase retrieval. However, unlike our problem, these works all focus symmetric matrices and operate in the low-dimensional regime where sample size is more than the parameter size.…”

Section: Related Workmentioning

confidence: 99%

Generalization Guarantees for Neural Architecture Search with Train-Validation Split

Oymak¹,

Li²,

Soltanolkotabi³

2021

Preprint

View full text Add to dashboard Cite

Neural Architecture Search (NAS) is a popular method for automatically designing optimized architectures for high-performance deep learning. In this approach, it is common to use bilevel optimization where one optimizes the model weights over the training data (lower-level problem) and various hyperparameters such as the configuration of the architecture over the validation data (upper-level problem). This paper explores the statistical aspects of such problems with train-validation splits. In practice, the lower-level problem is often overparameterized and can easily achieve zero loss. Thus, a-priori it seems impossible to distinguish the right hyperparameters based on training loss alone which motivates a better understanding of the role of train-validation split. To this aim this work establishes the following results:• We show that refined properties of the validation loss such as risk and hyper-gradients are indicative of those of the true test loss. This reveals that the upper-level problem helps select the most generalizable model and prevent overfitting with a near-minimal validation sample size. Importantly, this is established for continuous spaces -which are highly relevant for popular differentiable search schemes.• We establish generalization bounds for NAS problems with an emphasis on an activation search problem. When optimized with gradient-descent, we show that the train-validation procedure returns the best (model, architecture) pair even if all architectures can perfectly fit the training data to achieve zero error.• Finally, we highlight rigorous connections between NAS, multiple kernel learning, and low-rank matrix learning. The latter leads to novel algorithmic insights where the solution of the upper problem can be accurately learned via efficient spectral methods to achieve near-minimal risk.

show abstract

Section: Related Workmentioning

confidence: 99%

Generalization Guarantees for Neural Architecture Search with Train-Validation Split

Oymak¹,

Li²,

Soltanolkotabi³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Dong et al (2019), as well as studied theoretically, see e.g. Mondelli et al (2020). Warm initialization is formally needed to obtain non-trivial results for times linear in the dimension.…”

Section: The Analyzed Algorithmsmentioning

confidence: 99%

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Mignacco,

Urbani,

Zdeborová

2021

Preprint

View full text Add to dashboard Cite

In this paper we investigate how gradient-based algorithms such as gradient descent, (multi-pass) stochastic gradient descent, its persistent variant, and the Langevin algorithm navigate non-convex losslandscapes and which of them is able to reach the best generalization error at limited sample complexity. We consider the loss landscape of the high-dimensional phase retrieval problem as a prototypical highly non-convex example. We observe that for phase retrieval the stochastic variants of gradient descent are able to reach perfect generalization for regions of control parameters where the gradient descent algorithm is not. We apply dynamical mean-field theory from statistical physics to characterize analytically the full trajectories of these algorithms in their continuous-time limit, with a warm start, and for large system sizes. We further unveil several intriguing properties of the landscape and the algorithms such as that the gradient descent can obtain better generalization properties from less informed initializations.

show abstract

“…The first phase of the artificial GAMP is designed so that its output vectors after T iterations (x T , ũT ) are close to the initialization (x 0 , u 0 ) of the true GAMP algorithm given by (3.1)-(3.2). This part of the algorithm is similar to the GAMP used in [MTV20] to approximate the spectral estimator xs . In particular, the state evolution recursion of the first phase (given in (B.2)) converges as T → ∞ to the following fixed point: lim…”

Section: Sketch Of the Proof Of Theoremmentioning

confidence: 99%

“…random matrix model and relies on a delicate decoupling argument between the outlier eigenvectors and the bulk. Here, we follow an approach developed in [MTV20], where a specially designed AMP is used to establish the joint empirical distribution of the signal, the spectral estimator, and the linear estimator.…”

Section: Introductionmentioning

confidence: 99%

Approximate Message Passing with Spectral Initialization for Generalized Linear Models

Mondelli¹,

Venkataramanan²

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the problem of estimating a signal from measurements obtained via a generalized linear model. We focus on estimators based on approximate message passing (AMP), a family of iterative algorithms with many appealing features: the performance of AMP in the high-dimensional limit can be succinctly characterized under suitable model assumptions; AMP can also be tailored to the empirical distribution of the signal entries, and for a wide class of estimation problems, AMP is conjectured to be optimal among all polynomial-time algorithms.However, a major issue of AMP is that in many models (such as phase retrieval), it requires an initialization correlated with the ground-truth signal and independent from the measurement matrix. Assuming that such an initialization is available is typically not realistic. In this paper, we solve this problem by proposing an AMP algorithm initialized with a spectral estimator. With such an initialization, the standard AMP analysis fails since the spectral estimator depends in a complicated way on the design matrix. Our main technical contribution is the construction and analysis of a two-phase artificial AMP algorithm that first produces the spectral estimator, and then closely approximates the iterates of the true AMP. Our analysis yields a rigorous characterization of the performance of AMP with spectral initialization in the high-dimensional limit. We also provide numerical results that demonstrate the validity of the proposed approach.

show abstract

Optimal Combination of Linear and Spectral Estimators for Generalized Linear Models

Cited by 5 publications

References 21 publications

Generalization Guarantees for Neural Architecture Search with Train-Validation Split

Generalization Guarantees for Neural Architecture Search with Train-Validation Split

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Approximate Message Passing with Spectral Initialization for Generalized Linear Models

Contact Info

Product

Resources

About