Multi-Agent Thompson Sampling for Bandit Applications with Sparse Neighbourhood Structures

Abstract

Multi-agent coordination is prevalent in many real-world applications. However, such coordination is challenging due to its combinatorial nature. An important observation in this regard is that agents in the real world often only directly affect a limited set of neighbouring agents. Leveraging such loose couplings among agents is key to making coordination in multi-agent systems feasible. In this work, we focus on learning to coordinate. Specifically, we consider the multi-agent multi-armed bandit framework, in which fully cooperative loosely-coupled agents must learn to coordinate their decisions to optimize a common objective. We propose multi-agent Thompson sampling (MATS), a new Bayesian exploration-exploitation algorithm that leverages loose couplings. We provide a regret bound that is sublinear in time and low-order polynomial in the highest number of actions of a single agent for sparse coordination graphs. Additionally, we empirically show that MATS outperforms the state-of-the-art algorithm, MAUCE, on two synthetic benchmarks, and a novel benchmark with Poisson distributions. An example of a loosely-coupled multi-agent system is a wind farm. Coordination within the wind farm is necessary to maximize power production. As upstream wind turbines only affect nearby downstream turbines, we can use MATS to efficiently learn the optimal control mechanism for the farm. To demonstrate the benefits of our method toward applications we apply MATS to a realistic wind farm control task. In this task, wind turbines must coordinate their alignments with respect to the incoming wind vector in order to optimize power production. Our results show that MATS improves significantly upon state-of-the-art coordination methods in terms of performance, demonstrating the value of using MATS in practical applications with sparse neighbourhood structures.

Scalable Optimization for Wind Farm Control using Coordination Graphs

“…To ensure scalability of the control optimization process toward large wind farms, we need a learning algorithm that can decompose and exploit the farm's topology. To this end, we propose a new wind farm control algorithm, Set-Point Thompson Sampling (SPTS), which constructs a dependency graph and regimes from wake fields and load profiles, and relies on multi-agent Thompson sampling (MATS) [40,41] to efficiently explore the joint set-point configuration space using the factored representation.…”

“…The expected power productions P w s G(w) for each possible local set-point configuration s G(w) are unknown. Similar to MATS [40], SPTS uses a Bayesian formalism, which means users can exert their beliefs over P w s G(w) in the form of a prior. If wind turbines were not affected by wake, the incoming wind speed can be used to predict the expected power production of the wind turbine, which is provided with the turbine design specifications [28].…”

Context Aware Control Systems: An Engineering Applications Perspective

“…Distributed control seem to cope well with variations in operating conditions of power systems [133], while multiobjective optimization may offer versatility in short-and long-term economic objectives [134]. Multi-agent artificial intelligence algorithms applied to wind farm throughput optimization seems to be better positioned when data and structural features are integrated [135].…”

Asymptotically Optimal Bandits under Weighted Information

“…These extensions are motivated by problems arising in various fields, including online recommendation systems (search engines, display ads, etc. ), financial portfolio design [4,5], adaptive network routing [6], web crawling [7], clinical trials [8,1], and, recently, in control theory [9] and system identification [10,11].…”