Algorithms for subset selection in linear regression

Abstract

We study the problem of selecting a subset of k random variables to observe that will yield the best linear prediction of another variable of interest, given the pairwise correlations between the observation variables and the predictor variable. Under approximation preserving reductions, this problem is also equivalent to the "sparse approximation" problem of approximating signals concisely.We propose and analyze exact and approximation algorithms for several special cases of practical interest. We give an FPTAS when the covariance matrix has constant bandwidth, and exact algorithms when the associated covariance graph, consisting of edges for pairs of variables with non-zero correlation, forms a tree or has a large (known) independent set. Furthermore, we give an exact algorithm when the variables can be embedded into a line such that the covariance decreases exponentially in the distance, and a constant-factor approximation when the variables have no "conditional suppressor variables".Much of our reasoning is based on perturbation results for the R 2 multiple correlation measure, frequently used as a measure for "goodness-of-fit statistics". It lies at the core of our FPTAS, and also allows us to extend exact algorithms to approximation algorithms when the matrix "nearly" falls into one of the above classes. We also use perturbation analysis to prove approximation guarantees for the widely used "Forward Regression" heuristic when the observation variables are nearly independent.

Optimal experimental design for sampling voltage on dendritic trees in the low-SNR regime

“…defined above is submodular (Das and Kempe, 2008;Krause et al, 2008a), we found empirically that in our case ρ(·) is not quite submodular (see section 4.4 for details). Nevertheless, the greedy algorithm (and the lazy implementation) proved to be quite effective in practice, as we will see below.…”

“…Selecting the optimal set of observations O is NP-hard in general (Das and Kempe, 2008;Krause et al, 2008a), and therefore we must rely on approximate methods to design an optimal sampling scheme. There is a significant body of work on maximizing objective functions that are submodular (see, e.g., (Krause et al, 2008b) and references therein).…”

“…Using these computed covariance matrices we can easily calculate the expected mean-squared error, the mutual information, and other design metrics. To efficiently search the space of possible designs, we utilized "lazy greedy" methods from the literature on submodular optimization (Nemhauser et al, 1978;Krause et al, 2007Krause et al, , 2008bDas and Kempe, 2008). These lazy evaluation methods proved critical to making the optimization tractable.…”

AMBROSia: An Autonomous Model-Based Reactive Observing System

“…Over the past year, we have obtained the following key results (which are currently under submission [1] The quality of approximation is characterized precisely in [1], but omitted here due to space constraints. This result improves on the ones of [6,17], in that it analyzes a more commonly used algorithm, and obtains somewhat improved bounds.…”

“…Hence, we also study the case of sensors embedded in a metric space, where the covariance between sensors' readings is a monotone decreasing function of their distance. The general version of this problem is the subject of ongoing work, but [1] contains a promising initial finding:…”