Learning in Congestion Games with Bandit Feedback

Abstract

Learning Nash equilibria is a central problem in multi-agent systems. In this paper, we investigate congestion games, a class of games with benign theoretical structure and broad real-world applications. We first propose a centralized algorithm based on the optimism in the face of uncertainty principle for congestion games with (semi-)bandit feedback, and obtain finite-sample guarantees. Then we propose a decentralized algorithm via a novel combination of the Frank-Wolfe method and G-optimal design. By exploiting the structure of the congestion game, we show the sample complexity of both algorithms depends only polynomially on the number of players and the number of facilities, but not the size of the action set, which can be exponentially large in terms of the number of facilities. We further define a new problem class, Markov congestion games, which allows us to model the non-stationarity in congestion games. We propose a centralized algorithm for Markov congestion games, whose sample complexity again has only polynomial dependence on all relevant problem parameters, but not the size of the action set.

Instance-dependent Sample Complexity Bounds for Zero-sum Matrix Games

“…Additionally, Dou et al (2022) provide sample complexity results for the centralized and decentralized problem settings under both semi-bandit and bandit feedback. Similarly, Cui et al (2022) study the regret of the Nash Q-learning algorithm for two-player turn based Markov games, and introduce the first gap dependent logarithmic upper bounds, which match theoretical lower bounds up to log factors, in the episodic tabular setting.…”

Halftoning with Multi-Agent Deep Reinforcement Learning

“…In this part we will show the Nash-UCB algorithm proposed in Cui et al [2022] satisfies Assumption 1. We carry out the proof by pointing out the modifications we need to make in their proof.…”

“…This versatile approach 1.2 Related Work (Stationary) Multi-agent reinforcement learning. Numerous works have been devoted to learning equilibria in (stationary) multi-agent systems, including zero-sum Markov games [Bai et al, 2020, general-sum Markov games , Mao et al, 2022, Song et al, 2021, Daskalakis et al, 2022, Wang et al, 2023, Cui et al, 2023, Markov potential games [Leonardos et al, 2021, Song et al, 2021, Ding et al, 2022, Cui et al, 2023, congestion games [Cui et al, 2022], extensive-form games [Kozuno et al, 2021, and partially observable Markov games [Liu et al, 2022]. These works aim to learn equilibria with bandit feedback efficiently, measured by either regret or sample complexity.…”