On Motzkin’s method for inconsistent linear systems

Haddock, Jamie; Needell, Deanna

doi:10.1007/s10543-018-0737-6

Cited by 39 publications

(29 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the area of randomized methods for solving large linear system there is one particular method, the Kaczmarz-Motzkin algorithm [12,28] that can work as gossip algorithm with the same update as the GGE when is use to solve the homogeneous linear system with matrix the Incidence matrix of the network.…”

Section: Further Connections Between Methods For Solving Linear Systems and Gossip Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols

Loizou

Richtárik²

2021

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

In this work we present a new framework for the analysis and design of randomized gossip algorithms for solving the average consensus problem. We show how classical randomized iterative methods for solving linear systems can be interpreted as gossip algorithms when applied to special systems encoding the underlying network and explain in detail their decentralized nature. Our general framework recovers a comprehensive array of well-known gossip algorithms as special cases, including the pairwise randomized gossip algorithm and path averaging gossip, and allows for the development of provably faster variants. The flexibility of the new approach enables the design of a number of new specific gossip methods. For instance, we propose and analyze novel block and the first provably accelerated randomized gossip protocols, and dual randomized gossip algorithms.From a numerical analysis viewpoint, our work is the first that explores in depth the decentralized nature of randomized iterative methods for linear systems and proposes them as methods for solving the average consensus problem.We evaluate the performance of the proposed gossip protocols by performing extensive experimental testing on typical wireless network topologies. Keywords randomized gossip algorithms • average consensus • weighted average consensus • stochastic methods • linear systems • randomized Kaczmarz • randomized block Kaczmarz • randomized coordinate descent • heavy ball momentum • Nesterov's acceleration • duality • convex optimization • wireless sensor networks Mathematical Subject Classifications 93A14 • 68W15 • 68Q25 • 68W20 • 68W40 • 65Y20 • 90C15 • 90C20 • 90C25 • 15A06 • 15B52 • 65F10 * Part of the results was presented in [46, 49, 45] 1

show abstract

Section: Further Connections Between Methods For Solving Linear Systems and Gossip Algorithmsmentioning

confidence: 99%

“…and recall that x k+1 = I − A C: (A C: A C: ) † A C: x k is the update rule of the standard RBK (28). Thus, in analogy to the standard RBK, in the k th step, a random set of edges is selected and q ≤ n connected components are formed as a result.…”

Section: Randomized Block Kaczmarz Gossip With Heavy Ball Momentummentioning

confidence: 99%

Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols

Loizou

Richtárik²

2021

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

show abstract

“…In this paper, we consider the randomized Kaczmarz (RK) algorithm, the randomized Gauss–Seidel (RGS) algorithm, the randomized extended Kaczmarz (REK) algorithm, and the randomized extended Gauss–Seidel (REGS) algorithm . Convergence rates of these algorithms (including their deterministic variants) have been considered extensively; see, for example, other works . Let A † denote the Moore–Penrose pseudoinverse of A .…”

Section: Introductionmentioning

confidence: 99%

“…11 Convergence rates of these algorithms (including their deterministic variants) have been considered extensively; see, for example, other works. 1,2,5,6,11,13,[20][21][22][23][24][25][26] Let A † denote the Moore-Penrose pseudoinverse 27 of A. We summarize the convergence of RK, RGS, REK, and REGS in expectation to the Moore-Penrose pseudoinverse solution A † b for all types of linear systems in Table 1.Main contributions.…”

mentioning

confidence: 99%

Tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss–Seidel algorithms

2019

Numerical Linear Algebra App

View full text Add to dashboard Cite

Summary The randomized extended Kaczmarz and Gauss–Seidel algorithms have attracted much attention because of their ability to treat all types of linear systems (consistent or inconsistent, full rank or rank deficient). In this paper, we present tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss–Seidel algorithms. Numerical experiments are given to illustrate the theoretical results.

show abstract

“…This so-called Motzkin's method has also been known as the Kaczmarz method with the most violated constraint or maximal-residual control ( [13], [12], [2]). It is intuitive to expect that Motzkin's method converges more efficiently than Kaczmarz per iteration (although each iteration is clearly much more expensive, as one has to swipe over all m equations to determine the best one to satisfy (I.4)) A step towards the theoretical guarantee of the convergence of Motzkin's method improved convergence rate was made in [7], where the authors have shown that ∞ is a so-called dynamic range of the system.…”

Section: Introductionmentioning

confidence: 99%

Sketching for Motzkin’s Iterative Method for Linear Systems

Rebrova

Needell

2019

2019 53rd Asilomar Conference on Signals, Systems, and Computers

Self Cite

View full text Add to dashboard Cite

Projection-based iterative methods for solving large over-determined linear systems are well-known for their simplicity and computational efficiency. It is also known that the correct choice of a sketching procedure (i.e., preprocessing steps that reduce the dimension of each iteration) can improve the performance of iterative methods in multiple ways, such as, to speed up the convergence of the method by fighting inner correlations of the system, or to reduce the variance incurred by the presence of noise. In the current work, we show that sketching can also help us to get better theoretical guarantees for the projection-based methods. Specifically, we use good properties of Gaussian sketching to prove an accelerated convergence rate of the sketched relaxation (also known as Motzkins) method. The new estimates hold for linear systems of arbitrary structure. We also provide numerical experiments in support of our theoretical analysis of the sketched relaxation method.

show abstract

On Motzkin’s method for inconsistent linear systems

Cited by 39 publications

References 15 publications

Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols

Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols

Tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss–Seidel algorithms

Sketching for Motzkin’s Iterative Method for Linear Systems

Contact Info

Product

Resources

About