Orthonormal Sketches for Secure Coded Regression

Abstract

In this work, we propose methods for speeding up linear regression distributively, while ensuring security. We leverage randomized sketching techniques, and improve straggler resilience in asynchronous systems. Specifically, we apply a random orthonormal matrix and then subsample blocks, to simultaneously secure the information and reduce the dimension of the regression problem. In our setup, the transformation corresponds to an encoded encryption in an approximate gradient coding scheme, and the subsampling corresponds to the responses of the non-straggling workers; in a centralized coded computing network. This results in a distributive iterative sketching approach for an ℓ2-subspace embedding, i.e. a new sketch is considered at each iteration. We also focus on the special case of the Subsampled Randomized Hadamard Transform, which we generalize to block sampling; and discuss how it can be modified in order to secure the data.

Weighted Gradient Coding with Leverage Score Sampling

“…Let β = 1. For δ = e Θ(1) /(N d), the sketches of Algorithm 1 and the block-SRHT of [56] achieve the same asymptotic ℓ 2 -s.e. guarantee, for the same number of sampling trials q.…”

“…Finally, in Proposition 5 we show when the block leverage score sampling sketch of Algorithm 1 and the block-SRHT of [56] have the same ℓ 2 -s.e. guarantee.…”

“…Multiplying the data by random matrix spreads the information in the rows of the matrix, such that all rows are of equal importance; and the new matrix is "incoherent". In Appendix C, we show when Algorithm 1 and the corresponding block sampling counterpart of the SRHT [56] achieve the same asymptotic guarantees, for the same number of sampling trials q.…”

“…An alternative view point is that the matrix Ũexp := Ψ • (ΣV ⊤ ) −1 has uniform Frobenius block scores, which further justifies the proposed GC approach for homogeneous servers. This resembles the main idea behind the SRHT [43], [53] and different variants known as block-SRHT [56], [73], where a random projection is applied to the data matrix to "flatten" its leverage scores, i.e. make them close to uniform.…”