Solving Non-Convex Non-Differentiable Min-Max Games Using Proximal Gradient Method

Abstract

Min-max saddle point games appear in a wide range of applications in machine leaning and signal processing. Despite their wide applicability, theoretical studies are mostly limited to the special convex-concave structure. While some recent works generalized these results to special smooth non-convex cases, our understanding of non-smooth scenarios is still limited. In this work, we study special form of non-smooth min-max games when the objective function is (strongly) convex with respect to one of the player's decision variable. We show that a simple multi-step proximal gradient descent-ascent algorithm converges to -first-order Nash equilibrium of the min-max game with the number of gradient evaluations being polynomial in 1/ . We will also show that our notion of stationarity is stronger than existing ones in the literature. Finally, we evaluate the performance of the proposed algorithm through adversarial attack on a LASSO estimator.Keywords-Non-convex min-max games, First-order Nash equilibria, Proximal gradient descent ascent IntroductionNon-convex min-max saddle point games appear in a wide range of applications such as training Generative Adversarial Networks [1,2,3,4], fair statistical inference [5,6,7], and training robust neural networks and systems [8,9,10]. In such a game, the goal is to solve the optimization problem of the formwhich can be considered as a two player game where one player aims at increasing the objective, while the other tries to minimize the objective. Using game theoretic point of view, we may aim for finding Nash equilibria [11] in which no player can do better off by unilaterally changing its strategy. Unfortunately, finding/checking such Nash equilibria is hard in general [12] for non-convex objective functions. Moreover, such Nash equilibria might not even exist. Therefore, many works focus on special cases such as convex-concave problems where f (θ, .) is concave for any given θ and f (., α) is convex for any given α. Under this assumption, different algorithms such as optimistic mirror descent [13,14,15,16], Frank-Wolfe algorithm [17,18] and Primal-Dual method [19] have been studied.In the general non-convex settings, [20] considers the weakly convex-concave case and proposes a primal-dual based approach for finding approximate stationary solutions. More recently, the research works [21,22,23,24] * This arXiv submission includes the details of the proofs for the paper accepted for publication in the proceeding of the 45 th International Conference on Acoustics, Speech, and Signal Processing (ICASSP). arXiv:2003.08093v1 [math.OC] 18 Mar 2020examine the min-max problem in non-convex-(strongly)-concave cases and proposed first-order algorithms for solving them. Some of the results have been accelerated in the "Moreau envelope regime" by the recent interesting work [25]. This work first starts by studying the problem in smooth strongly convex-concave and convex-concave settings, and proposes an algorithm based on the combination of Mirror-Prox [26] and Nesterov's accelerated gra...

A Max-Min Entropy Framework for Reinforcement Learning

“…At the early stage of learning starting with random Q-network weight initialization and random policy-network weight initialization, there is little Q-value difference with respect to either state or action, as seen in Figs. 2(b) and 2(d), so the entropy term is dominant in the policy update (6) and the policy entropy increases with the policy distribution approaching the uniform distribution, as seen in Fig. 2(c).…”

“…Indeed, it is seen in Fig. 1(c) that (6) does not affect the policy update, only the second term H(π(•|s t )) works, and thus the policy update yields π to converge to the uniform policy for every state maximizing the total entropy. Hence, the exploration radius in the case of α Q = 0 is almost the same as that of the uniform policy, as seen in Fig.…”

“…Under the convex-concave assumption, there exist many algorithms to find the solution to a max-min problem by using optimistic mirror descent [45], Frank-Wolfe algorithm [18], and Primal-Dual method [25]. However, non-convex max-min problems are more challenging [38] and there are several recent studies to find (approximiate) solutions to non-convex max-min optimization problems [6,42,44]. This framework has been used in various optimization/control domains: fair resource allocation [32], inference [4,57], generative adversarial network (GAN) [2,19], robust training [34], and reinforcement learning [55].…”

“…Furthermore, the increasing true soft Q-value of soft policy iteration also contributes to Q-value increase. Then, by the policy update (6), the policy is learned to visit the states with high Q-values near the starting point more frequently. This is evident in the lower row of Fig.…”

Solving a Class of Non-Convex Min-Max Games Using Adaptive Momentum Methods

“…However, computing a Nash equilibrium point is NP-hard in general [22,19], and it may not even exist [23]. As a result, since we are considering the general non-convexnon-concave regime, we settle in computing a first-order Nash equilibrium point [24,25] defined next.…”