Lower Bounds and a Near-Optimal Shrinkage Estimator for Least Squares Using Random Projections

Sridhar, Srivatsan; Pilancı, Mert; Özgür, Ayfer

doi:10.1109/jsait.2020.3039509

Cited by 3 publications

(4 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The proof of Lemma 2.9 is based on Fisher information and Cramér-Rao lower bound while Lemma 2.10 additionally employs the van Trees inequality. We refer the reader to [28] for details on these single sketch lower bounds.…”

Section: Error Lower Bounds Via Fisher Informationmentioning

confidence: 99%

“…Lemma 2.10 (General estimators, [28]): For any single sketch estimator x, which is possibly biased, obtained from the Gaussian sketched data SA and Sb, the expected error is lower bounded as follows…”

Section: Error Lower Bounds Via Fisher Informationmentioning

confidence: 99%

“…Hence, we can leverage the lower bound result for the single sketch estimator. We now give the details of how to find the lower bound for the single sketch estimator which was first shown in [28].…”

Section: A Proofs Of Theorems and Lemmas In Section IImentioning

confidence: 99%

See 2 more Smart Citations

Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration and Lower Bounds

Bartan¹,

Pilancı²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We consider distributed optimization methods for problems where forming the Hessian is computationally challenging and communication is a significant bottleneck. We leverage randomized sketches for reducing the problem dimensions as well as preserving privacy and improving straggler resilience in asynchronous distributed systems. We derive novel approximation guarantees for classical sketching methods and establish tight concentration results that serve as both upper and lower bounds on the error. We then extend our analysis to the accuracy of parameter averaging for distributed sketches. Furthermore, we develop unbiased parameter averaging methods for randomized second order optimization for regularized problems that employ sketching of the Hessian. Existing works do not take the bias of the estimators into consideration, which limits their application to massively parallel computation. We provide closed-form formulas for regularization parameters and step sizes that provably minimize the bias for sketched Newton directions. Additionally, we demonstrate the implications of our theoretical findings via large scale experiments on a serverless cloud computing platform.

show abstract

Section: Error Lower Bounds Via Fisher Informationmentioning

confidence: 99%

Section: Error Lower Bounds Via Fisher Informationmentioning

confidence: 99%

See 1 more Smart Citation

Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration and Lower Bounds

Bartan¹,

Pilancı²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Lastly, we mention another sketch-and-solve approach which computes x : = argmin x 1 2 SAx − Sb 2 2 + λ 2 x 2 2 (see e.g. [15,40,39,46,14,7]). In [3], the authors showed that for m ≈ d e /ε, the estimate x satisfies f ( x)…”

mentioning

confidence: 99%

Fast Convex Quadratic Optimization Solvers with Adaptive Sketching-based Preconditioners

Lacotte¹,

Pilancı²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We consider least-squares problems with quadratic regularization and propose novel randomized iterative solvers with a sketching-based preconditioner and an adaptive embedding dimension. Without regularization, iterative methods are guaranteed to converge linearly provided that the sketch size scales in terms of the rank. In contrast, with regularization, the sketch size must scale in terms of the effective dimension of the data matrix to guarantee linear convergence. This yields computational savings as the effective dimension is always smaller than the rank, and eventually much smaller for data matrices with fast spectral decay. However, a major difficulty in choosing the sketch size in terms of the effective dimension lies in the fact that the latter is usually unknown in practice. Current sketching-based solvers for regularized least-squares fall short on addressing this issue. Our main contribution is to propose adaptive versions of standard sketching-based iterative solvers, namely, the iterative Hessian sketch and the preconditioned conjugate gradient method, that do not require a priori estimation of the effective dimension. We propose an adaptive mechanism to control the sketch size according to the progress made in each step of the iterative solver. If enough progress is not made, the sketch size increases to improve the convergence rate. We prove that the adaptive sketch size scales at most in terms of the effective dimension, and that our adaptive methods are guaranteed to converge linearly. Consequently, our adaptive methods improve the state-of-theart complexity for solving dense, ill-conditioned least-squares problems. Importantly, we illustrate numerically on several synthetic and real datasets that our method is extremely efficient and is often significantly faster than standard least-squares solvers such as a direct factorization based solver, the conjugate gradient method and its preconditioned variants.

show abstract

Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration, and Lower Bounds

Bartan

Pilancı

2023

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

Lower Bounds and a Near-Optimal Shrinkage Estimator for Least Squares Using Random Projections

Cited by 3 publications

References 28 publications

Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration and Lower Bounds

Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration and Lower Bounds

Fast Convex Quadratic Optimization Solvers with Adaptive Sketching-based Preconditioners

Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration, and Lower Bounds

Contact Info

Product

Resources

About