On the Covariance of Regression Coefficients

Abstract

In many applications, such as in multivariate meta-analysis or in the construction of multivariate models from summary statistics, the covariance of regression coefficients needs to be calculated without having access to individual patients' data. In this work, we derive an alternative analytic expression for the covariance matrix of the regression coefficients in a multiple linear regression model. In contrast to the well-known expressions which make use of the cross-product matrix and hence require access to individual data, we express the covariance matrix of the regression coefficients directly in terms of covariance matrix of the explanatory variables. In particular, we show that the covariance matrix of the regression coefficients can be calculated using the matrix of the partial correlation coefficients of the explanatory variables, which in turn can be calculated easily from the correlation matrix of the explanatory variables. This is very important since the covariance matrix of the explanatory variables can be easily obtained or imputed using data from the literature, without requiring access to individual data. Two important applications of the method are discussed, namely the multivariate meta-analysis of regression coefficients and the so-called synthesis analysis, and the aim of which is to combine in a single predictive model, information from different variables. The estimator proposed in this work can increase the usefulness of these methods providing better results, as seen by application in a publicly available dataset. Source code is provided in the Appendix and in http://www.compgen.org/tools/regression.

Estimating winter balance and its uncertainty from direct measurements of snow depth and density on alpine glaciers

“…We represent the uncertainty due to interpolation/extrapolation of gridcell-averaged b w in different ways for LR and OK. LR interpolation uncertainty is represented by a multivariate normal distribution of possible regression coefficients ( β ). The standard deviation of each distribution is calculated using the covariance of β as outlined in Bagos and Adam (2015), which ensures that the β values are internally consistent. The β distributions are randomly sampled and used to calculate gridcell-estimated b w .…”

Unveil the unseen: Exploit information hidden in noise

“…Depending on the machine learning algorithm, different techniques can be employed to estimate the uncertainty in prediction, here we highlight two approaches. Firstly, linear regression [39] and Gaussian processes [40] intrinsically compute the covariance matrix [34] from the training data and use it to estimate the uncertainty.…”

Making biased but better predictions: The trade-offs strategists face when they learn and use heuristics

“…A standard result in statistics is that the variance of an OLS estimator β true^ j increases in error variance σ 2 and decreases in sample size n (see, for example, Wooldridge, 2013: 93; a closed-form derivation of the elements in the covariation matrix V a r [ β j ] can be found in Bagos and Adam, 2015). Using this result, we can conclude that the upper bound of b O from equation (10) monotonously increases in ∥ β 2 − β 1 ∥ , monotonously increases in n (holding m fixed), and monotonously decreases in σ 2 (for each holding everything else constant).…”