Designing Statistical Estimators That Balance Sample Size, Risk, and Computational Cost

Abstract

Abstract-This paper proposes a tradeoff between computational time, sample complexity, and statistical accuracy that applies to statistical estimators based on convex optimization. When we have a large amount of data, we can exploit excess samples to decrease statistical risk, to decrease computational cost, or to trade off between the two. We propose to achieve this tradeoff by varying the amount of smoothing applied to the optimization problem. This work uses regularized linear regression as a case study to argue for the existence of this tradeoff both theoretically and experimentally. We also apply our method to describe a tradeoff in an image interpolation problem.Index Terms-Smoothing methods, statistical estimation, convex optimization, regularized regression, image interpolation, resource tradeoffs I. MOTIVATION M ASSIVE DATA presents an obvious challenge to statistical algorithms. We expect that the computational effort needed to process a data set increases with its size. The amount of computational power available, however, is growing slowly relative to sample sizes. As a consequence, large-scale problems of practical interest require increasingly more time to solve. This creates a demand for new algorithms that offer better performance when presented with large data sets.While it seems natural that larger problems require more effort to solve, Shalev-Shwartz and Srebro [1] showed that their algorithm for learning a support vector classifier actually becomes faster as the amount of training data increases. This and more recent works support an emerging viewpoint that treats data as a computational resource. That is, we should be able to exploit additional data to improve the performance of statistical algorithms.We consider statistical problems solved through convex optimization and propose the following approach: We can smooth statistical optimization problems more and more aggressively as the amount of available data increases. By controlling the amount of smoothing, we can exploit the additional data to decrease statistical risk, decrease computational cost, or trade off between the two. Our prior work [2] examined a similar time-data tradeoff achieved by applying a dual-smoothing method to (noiseless) regularized linear inverse problems. This paper generalizes those results, allowing for noisy measurements. The result is a tradeoff in computational time, sample size, and statistical accuracy.We use regularized linear regression problems as a specific example to illustrate our principle. We provide theoretical and numerical evidence that supports the existence of a time-data tradeoff achievable through aggressive smoothing of convex optimization problems in the dual domain. Our realization of the tradeoff relies on recent work in convex geometry that allows for precise analysis of statistical risk. In particular, we recognize the work done by Amelunxen et al.[3] to identify phase transitions in regularized linear inverse problems and the extension to noisy problems by Oymak and Hassibi [4]. While we...

Joint Activity Detection and Channel Estimation for IoT Networks: Phase Transition and Computation-Estimation Tradeoff

“…The authors of [39] showed that by modifying the original iterations, it is possible to achieve faster convergence rates to maintain the estimation accuracy without increasing the computational cost of each iteration considerably. More generally, smoothing techniques such as convex relaxation [40] or simply adding a nice smooth function to smooth the nondifferentiable objective function [41], [35], [42] often achieves a faster convergence rate. However, the amount of smoothing should be chosen carefully to guarantee the performance of sporadic device activity detection in IoT networks.…”

“…2) Computation and Estimation Trade-offs: To address the computational challenges in massive IoT networks with a limited time budget, we adopt the smoothing method to smooth the non-differentiable group sparsity inducing regularizer to accelerate the convergence rates. The computational speedups can be achieved by projecting onto simpler sets [40], varying the amount of smoothing [42], or adjusting the step sizes [38] applied to the optimization algorithms. However, the computational speedups will normally reduce the estimation accuracy.…”

“…Adding a smooth function to "smooth" the nondifferentiable objective function is a well-known idea in the context of sparse optimization, which makes the regularized problem easy to solve [41], [42]. In particular, for problem P, we augment R(Θ) by adding a smoothing function µ 2 Θ 2 F , where µ is a positive scalar and called as the smoothing parameter.…”

Statistical and computational tradeoff in genetic algorithm-based estimation

“…Wang et al [4], in a Sparse Principal Component Analysis framework, addressed the question of whether is possible to find an estimator that is computable in polynomial time, and then they analyzed its minimax optimal rate of convergence. Several other applications can be found in [5][6][7][8][9].…”

Multi denoising approximate message passing for optimal recovery with lower computational cost

“…Another work [5] examined the time-data trade-off for image interpolation problem, by varying the amount of smoothing applied to the convex optimization problem. However, most of these works have addressed the problem of computational and statistical trade-off in terms of modifying or improving a statistical optimization scheme, while in this work we pursue a different approach based on building a more general family of denoisers.…”