A fast randomized algorithm for overdetermined linear least-squares regression

Rokhlin, Vladimir; Tygert, Mark

doi:10.1073/pnas.0804869105

Cited by 161 publications

(222 citation statements)

References 4 publications

(2 reference statements)

Supporting

Mentioning

210

Contrasting

Unclassified

Order By: Relevance

“…In this section, we describe a fast method (presented in [13], [14]) for the generation of random orthogonal transformations and their application to arbitrary vectors.…”

Section: Pseudorandom Orthogonal Transformationsmentioning

confidence: 99%

See 1 more Smart Citation

A Randomized Approximate Nearest Neighbors Algorithm

Jones¹,

Osipov²,

Rokhlin³

2010

Self Cite

View full text Add to dashboard Cite

We present a randomized algorithm for the approximate nearest neighbor problem in ddimensional Euclidean space. Given N points {x j } in R d , the algorithm attempts to find k nearest neighbors for each of x j , where k is a user-specified integer parameter. The algorithm is iterative, and its CPU time requirements are proportional towith T the number of iterations performed. The memory requirements of the procedure are of the order N · (d + k). A byproduct of the scheme is a data structure, permitting a rapid search for the k nearest neighbors among {x j } for an arbitrary point x ∈ R d . The cost of each such query is proportional to, and the memory requirements for the requisite data structure are of the orderThe algorithm utilizes random rotations and a basic divide-and-conquer scheme, followed by a local graph search. We analyze the scheme's behavior for certain types of distributions of {x j }, and illustrate its performance via several numerical examples. A Randomized Approximate Nearest Neighbors AlgorithmPeter W. Jones † , Andrei Osipov ‡ , Vladimir Rokhlin ⋆ Research Report YALEU/DCS/RR-1434 Yale University September 14, 2010 † This author's research was supported in part by the DMS grant #0602635 and the ONR grants #N000140910108, #N000140910340; ‡ this author's research was supported in part by the AFOSR grant #FA9550-09-1-02-41; ⋆ this author's research was supported in part by the ONR grant #N00014-10-1-0570 and the AFOSR grant #FA9550-09-1-02-41.Approved for public release: distribution is unlimited. Keywords: Approximate nearest neighbors, randomized algorithms, fast random rotations 1 Report Documentation PageForm Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

show abstract

“…In this section, we describe a fast method (presented in [13], [14]) for the generation of random orthogonal transformations and their application to arbitrary vectors.…”

Section: Pseudorandom Orthogonal Transformationsmentioning

confidence: 99%

“…We have chosen to use random rotations in place of the usual random projections generated by selecting random Gaussian vectors. The fast random rotations require O(d · (log d)) operations, which is an improvement over methods using random projections (see [13], [14]). …”

Section: Introductionmentioning

confidence: 99%

A Randomized Approximate Nearest Neighbors Algorithm

Jones¹,

Osipov²,

Rokhlin³

2010

Self Cite

View full text Add to dashboard Cite

show abstract

“…Later, in 2008, Rokhlin and Tygert [22] described a related randomized algorithm for overdetermined systems. They used a randomized transform named SRFT that consists of m random Givens rotations, a random diagonal scaling, a discrete Fourier transform, and a random sampling of rows.…”

Section: Least Squares Solversmentioning

confidence: 99%

“…1 But it does not call the functions that return min-length solutions to rank-deficient or underdetermined systems. We choose Blendenpik out of several recently proposed randomized LS solvers, e.g., [22] and [3], because a high-performance implementation is publicly available and it is easy to adapt it to use multithreads. Blendenpik assumes that A has full rank.…”

Section: Numerical Experimentsmentioning

confidence: 99%

“…In recent years, however, motivated by large data problems, very nontrivial randomized algorithms for very large matrix problems have drawn considerable attention from researchers, originally in theoretical computer science and subsequently in numerical linear algebra and scientific computing. By randomized algorithms, we refer, in particular, to random sampling and random projection algorithms [8, 23, 9, 22, 2]. For a comprehensive overview of these developments, see the review of Mahoney [18], and for an excellent overview of numerical aspects of coupling randomization with classical low-rank matrix factorization methods, see the review of Halko, Martinsson, and Tropp [14].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

LSRN: A Parallel Iterative Solver for Strongly Over- or Underdetermined Systems

Meng

Saunders

Mahoney

2014

SIAM J. Sci. Comput.

111

127

View full text Add to dashboard Cite

We describe a parallel iterative least squares solver named that is based on random normal projection. computes the min-length solution to minx∈ℝn ‖Ax − b‖2, where A ∈ ℝm × n with m ≫ n or m ≪ n, and where A may be rank-deficient. Tikhonov regularization may also be included. Since A is involved only in matrix-matrix and matrix-vector multiplications, it can be a dense or sparse matrix or a linear operator, and automatically speeds up when A is sparse or a fast linear operator. The preconditioning phase consists of a random normal projection, which is embarrassingly parallel, and a singular value decomposition of size ⌈γ min(m, n)⌉ × min(m, n), where γ is moderately larger than 1, e.g., γ = 2. We prove that the preconditioned system is well-conditioned, with a strong concentration result on the extreme singular values, and hence that the number of iterations is fully predictable when we apply LSQR or the Chebyshev semi-iterative method. As we demonstrate, the Chebyshev method is particularly efficient for solving large problems on clusters with high communication cost. Numerical results show that on a shared-memory machine, is very competitive with LAPACK’s DGELSD and a fast randomized least squares solver called Blendenpik on large dense problems, and it outperforms the least squares solver from SuiteSparseQR on sparse problems without sparsity patterns that can be exploited to reduce fill-in. Further experiments show that scales well on an Amazon Elastic Compute Cloud cluster.

show abstract

Large‐scale inverse model analyses employing fast randomized data reduction

Lin

O’Malley

et al. 2017

Water Resources Research

View full text Add to dashboard Cite

When the number of observations is large, it is computationally challenging to apply classical inverse modeling techniques. We have developed a new computationally efficient technique for solving inverse problems with a large number of observations (e.g., on the order of 107 or greater). Our method, which we call the randomized geostatistical approach (RGA), is built upon the principal component geostatistical approach (PCGA). We employ a data reduction technique combined with the PCGA to improve the computational efficiency and reduce the memory usage. Specifically, we employ a randomized numerical linear algebra technique based on a so‐called “sketching” matrix to effectively reduce the dimension of the observations without losing the information content needed for the inverse analysis. In this way, the computational and memory costs for RGA scale with the information content rather than the size of the calibration data. Our algorithm is coded in Julia and implemented in the MADS open‐source high‐performance computational framework (http://mads.lanl.gov). We apply our new inverse modeling method to invert for a synthetic transmissivity field. Compared to a standard geostatistical approach (GA), our method is more efficient when the number of observations is large. Most importantly, our method is capable of solving larger inverse problems than the standard GA and PCGA approaches. Therefore, our new model inversion method is a powerful tool for solving large‐scale inverse problems. The method can be applied in any field and is not limited to hydrogeological applications such as the characterization of aquifer heterogeneity.

show abstract

A fast randomized algorithm for overdetermined linear least-squares regression

Cited by 161 publications

References 4 publications

A Randomized Approximate Nearest Neighbors Algorithm

A Randomized Approximate Nearest Neighbors Algorithm

LSRN: A Parallel Iterative Solver for Strongly Over- or Underdetermined Systems

Large‐scale inverse model analyses employing fast randomized data reduction

Contact Info

Product

Resources

About