Random projections for Bayesian regression

Abstract

This article deals with random projections applied as a data reduction technique for Bayesian regression analysis. We show sufficient conditions under which the entire $d$-dimensional distribution is approximately preserved under random projections by reducing the number of data points from $n$ to $k\in O(\operatorname{poly}(d/\varepsilon))$ in the case $n\gg d$. Under mild assumptions, we prove that evaluating a Gaussian likelihood function based on the projected data instead of the original data yields a $(1+O(\varepsilon))$-approximation in terms of the $\ell_2$ Wasserstein distance. Our main result shows that the posterior distribution of Bayesian linear regression is approximated up to a small error depending on only an $\varepsilon$-fraction of its defining parameters. This holds when using arbitrary Gaussian priors or the degenerate case of uniform distributions over $\mathbb{R}^d$ for $\beta$. Our empirical evaluations involve different simulated settings of Bayesian linear regression. Our experiments underline that the proposed method is able to recover the regression model up to small error while considerably reducing the total running time

Data Science: the impact of statistics

“…More advanced methods comprise functional regression for functional data [38], quantile regression [25], and regression based on loss functions other than squared error loss like, e.g., Lasso regression [11,21]. In the context of Big Data, the challenges are similar to those for classification methods given large numbers of observations n (e.g., in data streams) and / or large numbers of features p. For the reduction of n, data reduction techniques like compressed sensing, random projection methods [20] or sampling-based procedures [28] enable faster computations. For decreasing the number p to the most influential features, variable selection or shrinkage approaches like the Lasso [21] can be employed, keeping the interpretability of the features.…”

A Conceptual Framework for Lithium-ion Battery RUL Prediction Using Deep Learning

“…By using the training model, the lithium-ion batteries RUL could be predicted. To evaluate the performance of the deep neural network model, the prediction accuracy needs to compare with other approaches such as Bayesian Regression [16], the support vector machine (SVM) [17], Linear Regression [18] and etc. To represent the prediction accuracy, statistics based evaluation methods, e.g., standard deviation, mean squared error, root mean square error (RMSE), could be adopted to evaluate and compare the performance of different prediction models.…”

Coresets-Methods and History: A Theoreticians Design Pattern for Approximation and Streaming Algorithms

“…In particular, not only maximum likelihood estimators are approximated under random projections. Geppert et al [54] showed that in important classes of Bayesian regression models, the whole structure of the posterior distribution is preserved. This yields much faster algorithms for the widely applicable and flexible, but at the same time computationally demanding Bayesian machinery.…”

“…A parallel least squares regression solver LSRN was developed in [78,97]. An implementation of some of the presented sketching techniques named RaProR was made available for the statistics programming language R [53,54,87].…”

“…Some examples include clustering [4,17,42,50,75], classification [56,59,89], regression [34,40,41,54], and the smallest enclosing ball problem [6,15,16,48]; we refer to [84] for a recent extensive literature overview. If one of these base problems arises in some application, a coreset construction can often be used in a black box manner to improve the performance of the learning algorithm.…”