On the primal-dual dynamics of Support Vector Machines

Kosaraju, Krishna Chaitanya; Mohan, Shravan; Pasumarthy, Ramkrishna

doi:10.48550/arxiv.1805.00699

Cited by 2 publications

(3 citation statements)

References 13 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…to its efficiency in decentralized optimization in network applications [7]- [10]. Similar saddle point problems also appear naturally in the context of machine learning, e.g., in support vector machine representations [11] and in the adversarial training of deep networks [12].…”

Section: Introductionmentioning

confidence: 86%

Contraction and Robustness of Continuous Time Primal-Dual Dynamics

Nguyen

Turitsyn

et al. 2018

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

The Primal-Dual (PD) algorithm is widely used in convex optimization to determine saddle points. While the stability of the PD algorithm can be easily guaranteed, strict contraction is nontrivial to establish in most cases. This work focuses on continuous, possibly non-autonomous PD dynamics arising in a network context, in distributed optimization, or in systems with multiple time-scales. We show that the PD algorithm is indeed strictly contracting in specific metrics and analyze its robustness establishing stability and performance guarantees for different approximate PD systems. We derive estimates for the performance of multiple time-scale multi-layer optimization systems, and illustrate our results on a primal-dual representation of the Automatic Generation Control of power systems.

show abstract

Section: Introductionmentioning

confidence: 86%

Contraction and Robustness of Continuous Time Primal-Dual Dynamics

Nguyen

Turitsyn

et al. 2018

IEEE Control Syst. Lett.

View full text Add to dashboard Cite

show abstract

“…Our third formulation is the dual ascent algorithm (33). Substitution of the appropriate matrices into (33) leads to the centralized dual ascent dynamics 6 This follows since ker(E T ) = span(1n), and hence Im(E) is the subspace orthogonal to the vector 1n.…”

Section: A P P L I C Atmentioning

confidence: 99%

“…Recently, these algorithms have attracted renewed attention for e.g., in the context of machine learning [6], in the control literature for solving distributed convex optimization problems [7], where agents cooperate through a communication network to solve an optimization problem with minimal or no centralized coordination. Applications of distributed optimization include utility maximization [3], congestion management in communication networks [8], and control in power systems [9]- [14].…”

Section: Introductionmentioning

confidence: 99%

Input–Output Performance of Linear–Quadratic Saddle-Point Algorithms With Application to Distributed Resource Allocation Problems

Simpson-Porco

Poolla

Monshizadeh

et al. 2020

IEEE Trans. Automat. Contr.

View full text Add to dashboard Cite

Saddle-point or primal-dual methods have recently attracted renewed interest as a systematic technique to design distributed algorithms which solve convex optimization problems. When implemented online for streaming data or as dynamic feedback controllers, these algorithms become subject to disturbances and noise; convergence rates provide incomplete performance information, and quantifying input-output performance becomes more important. We analyze the input-output performance of the continuous-time saddle-point method applied to linearly constrained quadratic programs, providing explicit expressions for the saddle-point H2 norm under a relevant input-output configuration. We then proceed to derive analogous results for regularized and augmented versions of the saddlepoint algorithm. We observe some rather peculiar effectsa modest amount of regularization significantly improves the transient performance, while augmentation does not necessarily offer improvement. We then propose a distributed dual version of the algorithm which overcomes some of the performance limitations imposed by augmentation. Finally, we apply our results to a resource allocation problem to compare the inputoutput performance of various centralized and distributed saddlepoint implementations and show that distributed algorithms may perform as well as their centralized counterparts. I . I N T R O D U C T I O NSaddle-point methods are a class of continuous-time gradient-based algorithms for solving constrained convex optimization problems. Introduced in the early 1950s [1], [2], these algorithms are designed to seek the saddle points of the optimization problem's Lagrangian function. These saddle points are in one-to-one correspondence with the solutions of the first-order optimality (KKT) conditions, and the algorithm therefore drives its internal state towards the global optimizer of the convex program; see [3]-[5] for convergence results.Recently, these algorithms have attracted renewed attention for e.g., in the context of machine learning [6], in the control literature for solving distributed convex optimization problems [7], where agents cooperate through a communication network to solve an optimization problem with minimal or no centralized coordination. Applications of distributed optimization J. W. Simpson-Porco is with the include utility maximization [3], congestion management in communication networks [8], and control in power systems [9]- [14]. While most standard optimization algorithms require centralized information to compute the optimizer, saddle-point algorithms often yield distributed strategies in which agents perform state updates using only locally measured information and communication with some subset of other agents. We refer the reader to [4], [15]-[20] for control-theoretic interpretations of these algorithms.Rather than solve the optimization problem offline, it is desirable to run these distributed algorithms online as controllers, in feedback with system and/or disturbance measurements, to provide references so t...

show abstract

On the primal-dual dynamics of Support Vector Machines

Cited by 2 publications

References 13 publications

Contraction and Robustness of Continuous Time Primal-Dual Dynamics

Contraction and Robustness of Continuous Time Primal-Dual Dynamics

Input–Output Performance of Linear–Quadratic Saddle-Point Algorithms With Application to Distributed Resource Allocation Problems

Contact Info

Product

Resources

About