Coordinating Followers to Reach Better Equilibria: End-to-End Gradient Descent for Stackelberg Games

Abstract

A growing body of work in game theory extends the traditional Stackelberg game to settings with one leader and multiple followers who play a Nash equilibrium. Standard approaches for computing equilibria in these games reformulate the followers' best response as constraints in the leader's optimization problem. These reformulation approaches can sometimes be effective, but make limiting assumptions on the followers' objectives and the equilibrium reached by followers, e.g., uniqueness, optimism, or pessimism. To overcome these limitations, we run gradient descent to update the leader's strategy by differentiating through the equilibrium reached by followers. Our approach generalizes to any stochastic equilibrium selection procedure that chooses from multiple equilibria, where we compute the stochastic gradient by back-propagating through a sampled Nash equilibrium using the solution to a partial differential equation to establish the unbiasedness of the stochastic gradient. Using the unbiased gradient estimate, we implement the gradient-based approach to solve three Stackelberg problems with multiple followers. Our approach consistently outperforms existing baselines to achieve higher utility for the leader.

Differentiable Arbitrating in Zero-sum Markov Games

“…The incentive design could be reformulated as the Stackelberg game by treating the incentive design objective as the leading player. [86] proposed a gradient-decent algorithm to find NE for the Stackelberg bandit problem. [95] propose a value iteration method for solving the Stackelberg Markov game with convergence guarantee.…”

A Pricing Mechanism for Balancing the Charging of Ride-Hailing Electric Vehicle Fleets

“…Theorem 3: Let the Stackelberg game be defined as in (4) and its auxiliary best-response optimization problem as (19). Let the Assumptions 1, 2, 3 and 4 hold and {π t } be the sequence generated by the projected gradient descent method defined by equations ( 11), (12) and (13). Then it holds that lim t→+∞ J L (•, π t+1 ) − J L (•, π t ) = 0 , and every limit point of {π t } is stationary.…”

“…The proposed semi-decentralized approach leverages the existence of a Central aggregator typically required for decentralized computation of the Nash equilibrium of the game played between the followers [14]- [16]. Inspired by [13], [17], we show how the followers can locally compute the Jacobians by the means of Implicit function theorem, in an attempt to estimate the influence of the leader's decision variable on the attained Nash equilibrium of the followers. By designing an iterative procedure that concerns the central aggregator and the leading player and guarantees local improvement of the leader's objective at each iteration, we can provide formal convergence guarantees.…”

“…By designing an iterative procedure that concerns the central aggregator and the leading player and guarantees local improvement of the leader's objective at each iteration, we can provide formal convergence guarantees. Compared to [13], where a standing assumption was made on the conditions of the Implicit function theorem, here we explicitly address these conditions by proposing to differentiate the KKT conditions of an optimization problem that is equivalent to the standard best-response optimization problem of each follower. The complete schematic representation of the method is described in Figure 1.…”

“…In [12], the authors propose an iterative method for computing the local Stackelberg equilibrium but for the case of an unconstrained game. From the methodological point of view, our approach is, to a certain extent, a combination of [8] and [13]. The aggregative structure of the game in [8] matches the one presented in [6], [7], however, the assumed form of the leader's objective does not.…”