Inducing Equilibria via Incentives: Simultaneous Design-and-Play Finds Global Optima

Abstract

To regulate a social system comprised of self-interested agents, economic incentives (e.g., taxes, tolls, and subsidies) are often required to induce a desirable outcome. This incentive design problem naturally possesses a bi-level structure, in which an upperlevel "designer" modifies the payoffs of the agents with incentives while anticipating the response of the agents at the lower level, who play a non-cooperative game that converges to an equilibrium. The existing bi-level optimization algorithms developed in machine learning raise a dilemma when applied to this problem: anticipating how incentives affect the agents at equilibrium requires solving the equilibrium problem repeatedly, which is computationally inefficient; bypassing the time-consuming step of equilibriumfinding can reduce the computational cost, but may lead the designer to a sub-optimal solution. To address such a dilemma, we propose a method that tackles the designer's and agents' problems simultaneously in a single loop. In particular, at each iteration, both the designer and the agents only move one step based on the first-order information. In the proposed scheme, although the designer does not solve the equilibrium problem repeatedly, it can anticipate the overall influence of the incentives on the agents, which guarantees optimality. We prove that the algorithm converges to the global optima at a sublinear rate for a broad class of games.

Differentiable Arbitrating in Zero-sum Markov Games

“…More recently, incentive design has also been extended to the area of game theory and reinforcement learning. [54] utilize the incentive design in the multi-bandit problem and prove that the proposed algorithm converges to the global optimum at a sub-linear rate for a broad class of games. [64] provide an incentive-design mechanism for an uncooperative multi-agent system and optimize the upper-level incentive objective with Bayesian optimization, a sample-efficient optimization algorithm, instead of the gradient-based methods because the lower-level MARL problem is a black box.…”

Convergent First-Order Methods for Bi-level Optimization and Stackelberg Games

“…where f : X × Y −→ R and g : X × Y −→ R. Optimization problems of the form (1.1), referred as bi-level optimization, has been long-studied (Bard, 2013;Dempe, 2002;Vicente and Calamai, 1994). However due to its applications in variety of modern settings like deep learning (Liu et al, 2021b;Pedregosa, 2016), reinforcement learning (Das et al, 2021;Hong et al, 2020), adversarial learning (Lin et al, 2020;Liu et al, 2021a;Maheshwari et al, 2022), etc, which typically have non-linear, non-convex and high-dimensional structure, bi-level optimization problems have gained renewed interest (Colson et al, 2007;Sinha et al, 2017). Note that in the context of game theory, (1.1) also represents the solution to a two-player Stackelberg game (Li et al, 2022;Stackelberg et al, 1952;Yue and You, 2017).…”