Constrained Reinforcement Learning Has Zero Duality Gap

Paternain, Santiago; Chamon, Luiz F. O.; Calvo-Fullana, Miguel; Ribeiro, Alejandro

doi:10.48550/arxiv.1910.13393

Cited by 9 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• We show that adversarial training is equivalent to a stochastic optimization problem over a specific, non-atomic distribution, which we characterize using recent non-convex duality results [59,60]. Further, we show that a myriad of previous adversarial attacks reduce to particular, sub-optimal choices of this distribution.…”

Section: Introductionmentioning

confidence: 82%

“…and where f θ ∈ F is a fixed classifier. Roughly speaking, this problem is challenging due to the fact that we cannot compute the normalization constant in (59). Therefore, although the form of (59) indicates that the amount of mass placed on δ ∈ ∆ will be proportional to the loss (f θ (x, y)) when the data is perturbed by this perturbation δ, it's unclear how we can sample from this distribution in practice.…”

Section: E Deriving the Langevin Monte Carlo Samplermentioning

confidence: 99%

See 1 more Smart Citation

Adversarial Robustness with Semi-Infinite Constrained Learning

Robey¹,

Chamon²,

Pappas³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Despite strong performance in numerous applications, the fragility of deep learning to input perturbations has raised serious questions about its use in safety-critical domains. While adversarial training can mitigate this issue in practice, state-ofthe-art methods are increasingly application-dependent, heuristic in nature, and suffer from fundamental trade-offs between nominal performance and robustness. Moreover, the problem of finding worst-case perturbations is non-convex and underparameterized, both of which engender a non-favorable optimization landscape. Thus, there is a gap between the theory and practice of adversarial training, particularly with respect to when and why adversarial training works. In this paper, we take a constrained learning approach to address these questions and to provide a theoretical foundation for robust learning. In particular, we leverage semi-infinite optimization and non-convex duality theory to show that adversarial training is equivalent to a statistical problem over perturbation distributions, which we characterize completely. Notably, we show that a myriad of previous robust training techniques can be recovered for particular, sub-optimal choices of these distributions. Using these insights, we then propose a hybrid Langevin Monte Carlo approach of which several common algorithms (e.g., PGD) are special cases. Finally, we show that our approach can mitigate the trade-off between nominal and robust performance, yielding state-of-the-art results on MNIST and CIFAR-10. Our code is available at: https://github.com/arobey1/advbench.

show abstract

Section: Introductionmentioning

confidence: 82%

Section: E Deriving the Langevin Monte Carlo Samplermentioning

confidence: 99%

Adversarial Robustness with Semi-Infinite Constrained Learning

Robey¹,

Chamon²,

Pappas³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Theorem 2 There exists m * < ∞ such that as long as the number of branches m ≥ m * , the strong duality holds for the problem (16). Namely, P prl lin = D prl lin .…”

Section: Propositionmentioning

confidence: 99%

“…The convex duality also has important applications in machine learning. In [16], the design problem of an all-encompassing reward can be formulated as a constrained reinforcement learning problem, which is shown to have zero duality. This property gives a theoretical convergence guarantee of the primal-dual algorithm for solving this problem.…”

Section: Introductionmentioning

confidence: 99%

Parallel Deep Neural Networks Have Zero Duality Gap

Wang

Ergen

Pilancı

2021

Preprint

View full text Add to dashboard Cite

Training deep neural networks is a well-known highly non-convex problem. In recent work [17], it is shown that there is no duality gap for regularized two-layer neural networks with ReLU activation, which enables global optimization via convex programs. For multi-layer linear networks with vector outputs, we formulate convex dual problems and demonstrate that the duality gap is non-zero for depth three and deeper networks. However, by modifying the deep networks to more powerful parallel architectures, we show that the duality gap is exactly zero. Therefore, strong convex duality holds, and hence there exist equivalent convex programs that enable training deep networks to global optimality. We also demonstrate that the weight decay regularization in the parameters explicitly encourages low-rank solutions via closed-form expressions. For three-layer non-parallel ReLU networks, we show that strong duality holds for rank-1 data matrices, however, the duality gap is non-zero for whitened data matrices. Similarly, by transforming the neural network architecture into a corresponding parallel version, the duality gap vanishes.

show abstract

“…From the theoretical perspective, local and asymptotic convergence of various primal-dual algorithms have been studied in Chow et al (2017Chow et al ( , 2018; Tessler et al (2018); Yu et al (2019). Despite the objective being non-concave and the constraint set non-convex, it has been shown that the duality gap of a CMDP problem is zero (Altman, 1999;Paternain et al, 2019). Based on this property, the authors in Ding et al (2020) analyze a primal-dual natural policy gradient algorithm for a CMDP with a single constraint and present a finitetime convergence to global optimality in the tabular policy setting and with a general smooth policy class.…”

Section: Related Workmentioning

confidence: 99%

Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes

Zeng¹,

Doan²,

Romberg³

2021

Preprint

View full text Add to dashboard Cite

We consider a discounted cost constrained Markov decision process (CMDP) policy optimization problem, in which an agent seeks to maximize a discounted cumulative reward subject to a number of constraints on discounted cumulative utilities. To solve this constrained optimization program, we study an online actor-critic variant of a classic primal-dual method where the gradients of both the primal and dual functions are estimated using samples from a single trajectory generated by the underlying time-varying Markov processes. This online primal-dual natural actor-critic algorithm maintains and iteratively updates three variables: a dual variable (or Lagrangian multiplier), a primal variable (or actor), and a critic variable used to estimate the gradients of both primal and dual variables. These variables are updated simultaneously but on different time scales (using different step sizes) and they are all intertwined with each other. Our main contribution is to derive a finite-time analysis for the convergence of this algorithm to the global optimum of a CMDP problem. Specifically, we show that with a proper choice of step sizes the optimality gap and constraint violation converge to zero in expectation at a rate Op1{K 1{6 q, where K is the number of iterations. To our knowledge, this paper is the first to study the finite-time complexity of an online primal-dual actor-critic method for solving a CMDP problem. We also validate the effectiveness of this algorithm through numerical simulations.

show abstract

Constrained Reinforcement Learning Has Zero Duality Gap

Cited by 9 publications

References 0 publications

Adversarial Robustness with Semi-Infinite Constrained Learning

Adversarial Robustness with Semi-Infinite Constrained Learning

Parallel Deep Neural Networks Have Zero Duality Gap

Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processes

Contact Info

Product

Resources

About