Low rank approximation and regression in input sparsity time

Abstract

We design a new distribution over poly(rε −1 ) × n matrices S so that for any fixed n × d matrix A of rank r, with probability at least 9/10, SAx 2 = (1 ± ε) Ax 2 simultaneously for all x ∈ R d . Such a matrix S is called a subspace embedding. Furthermore, SA can be computed in O(nnz(A))time, where nnz(A) is the number of non-zero entries of A. This improves over all previous subspace embeddings, which required at least Ω(nd log d) time to achieve this property. We call our matrices S sparse embedding matrices.Using our sparse embedding matrices, we obtain the fastest known algorithms for overconstrained least-squares regression, low-rank approximation, approximating all leverage scores, and p-regression:for an n × d matrix A and an n × 1 column vector b, we obtain an algorithm running in O(nnz(A)) + O(d 3 ε −2 ) time, and another in O(nnz(A) log(1/ε)) +Õ(d 3 log(1/ε)) time.(HereÕ(f ) = f · log O(1) (f ).)• to obtain a decomposition of an n × n matrix A into a product of an n × k matrix L, a k × k diagonal matrix D, and an n × k matrix W , for whichwhere A k is the best rank-k approximation, our algorithm runs in O(nnz(A)) +Õ(nk 2 ε −4 + k 3 ε −5 ) time.• to output an approximation to all leverage scores of an n × d input matrix A simultaneously, with constant relative error, our algorithms run in O(nnz(A) log n) +Õ(r 3 ) time.• to output an x for whichfor an n×d matrix A and an n×1 column vector b, we obtain an algorithm running in O(nnz(A) log n)+ poly(rε −1 ) time, for any constant 1 ≤ p < ∞.We optimize the polynomial factors in the above stated running times, and show various tradeoffs. Finally, we provide preliminary experimental results which suggest that our algorithms are of interest in practice.

Toward a unified theory of sparse dimensionality reduction in Euclidean space

“…Meanwhile, our understanding for efficient Φ, such as the SJLT with small s, has not moved beyond the worst case. In some very specific examples of T we do have good bounds for settings of s, m that suffice, such as T the unit norm vectors in a d-dimensional subspace [CW13,MM13,NN13a], or all elements of T having small ∞ norm [Mat08,DKS10,KN10,BOR10]. However, our understanding for general T is non-existent.…”

“…Often an exact solution requires computing the singular value decomposition (SVD) of A, but using OSE's the running time is reduced to that for computing ΦA, plus computing the SVD of the smaller matrix ΦA. The work [CW13] showed s = 1 with small m is sufficient, yielding algorithms for least squares regression and low-rank approximation whose running times are linear in the number of non-zero entries in A for sufficiently lopsided rectangular matrices.…”

“…Linear subspace: Here T = {x ∈ E : x 2 = 1} for some d-dimensional linear subspace E ⊂ R n , for which achieving Equation (1.3) with m d/ε 2 is possible[AHK06,CW13]. A distribution satisfying Equation (1.3) for any d-dimensional subspace E is known as an oblivious subspace embedding (OSE).…”

Efficient inference of hidden Markov models from large observation sequences

“…We shall focus on sketching based algorithms in the tradition of Sarlos, 11 which use a small number of random linear combinations of columns and rows to approximate the SVD. [12][13][14] This further compression of the statistics matrix drives down the space complexity of the HMM inference problem at the expense of some small amount of error, which can be bounded analytically at the expense of some additional time and space. As the matrix is an approximation in the first place, this is likely not too great a cost.…”

Big data clustering via random sketching and validation

“…Dimensionality reduction and feature selection has been attracting a significant amount of attention [8], [16]. Recently popular approaches include the well-known "leverage-scores"-based approach [11], [12] and random projections (RP) [1], [3], [11], [12]. Leverage-scores-based approaches rely on the singular value decomposition of the data matrix to assign probabilities to each feature.…”