Numerical linear algebra in the streaming model

Abstract

We give near-optimal space bounds in the streaming model for linear algebra problems that include estimation of matrix products, linear regression, low-rank approximation, and approximation of matrix rank. In the streaming model, sketches of input matrices are maintained under updates of matrix entries; we prove results for turnstile updates, given in an arbitrary order. We give the first lower bounds known for the space needed by the sketches, for a given estimation error . We sharpen prior upper bounds, with respect to combinations of space, failure probability, and number of passes. The sketch we use for matrix A is simply S T A, where S is a sign matrix.Our results include the following upper and lower bounds on the bits of space needed for 1-pass algorithms. Here A is an n × d matrix, B is an n × d matrix, and c := d + d . These results are given for fixed failure probability; for failure probability δ > 0, the upper bounds require a factor of log(1/δ) more space. We assume the inputs have integer entries specified by O(log(nc)) bits, or O(log(nd)) bits. (Matrix Product) Output matrix C withWe show that Θ(c −2 log(nc)) space is needed. (Linear Regression) ForWe show that Θ(d 2 −1 log(nd)) space is needed.3. (Rank-k Approximation) Find matrixà k of rank no more than k, so thatwhere A k is the best rank-k approximation to A. Our lower bound is Ω(k −1 (n + d) log(nd)) space, and we give a one-pass algorithm matching this when A is given row-wise or column-wise. For general updates, we give a one-pass algorithm needing O(k −2 (n + d/ 2 ) log(nd)) space. We also give upper and lower bounds for algorithms using multiple passes, and a sketching analog of the CU R decomposition.

Almost Optimal Explicit Johnson-Lindenstrauss Families

“…Theorem 6 implies that any strong JL distribution can be derandomized using 2 log(1/δ)-wise independence giving an alternate proof of the derandomized JL result of Clarkson and Woodruff (Theorem 2.2 in [5]). This is because, by Markov's inequality with even, and for ε < 1,…”

“…On the other hand, despite much attention the best known explicit generators have seed-length at least min( Ω(log(1/δ) log d), Ω(log d + log 2 (1/δ)) ) [5], [11]. Besides being a natural problem in geometry as well as derandomization, an explicit JL generator with minimal randomness would likely help derandomize other geometric algorithms and metric embedding constructions.…”

“…Karnin et al [11] construct an explicit JL family with optimal target dimension and seed-length (1 + o(1)) log d + O(log 2 (1/(εδ))). Clarkson and Woodruff [5] showed that a random scaled sign matrix with O(log(1/δ))-wise independent entries satisfies the JL lemma, giving seed-length O(log(1/δ) log d). We make use of their result in our construction.…”

“…We then apply the result of Clarkson and Woodruff [5] and perform the final embedding into O(log(1/δ)/ε 2 ) dimensions by using a random scaled sign matrix with O(log(1/δ))-wise independent entries.…”

“…Theorem 1 shows the conditions for being a strong (d, s)-JL distribution are met by random Bernoulli matrices when 0 < ε ≤ 1, though in fact the conditions are also met for all ε > 0 (see the proof in [5] for example). Sometimes we omit the d, s terms in the notation above if these quantities are clear from context, or if it is not important to specify them.…”