Too Many cooks: Bayesian inference for coordinating Multi-agent Collaboration

Abstract

Collaboration requires agents to coordinate their behavior on the fly, sometimes cooperating to solve a single task together and other times dividing it up into sub-tasks to work on in parallel. Here, we develop Bayesian Delegation, a decentralized multi-agent learning mechanism with these abilities. Bayesian Delegation enables agents to rapidly infer the hidden intentions of others by inverse planning. We test Bayesian Delegation in a suite of multi-agent Markov decision processes inspired by cooking problems. On these tasks, agents with Bayesian Delegation coordinate both their high-level plans (e.g. what sub-task they should work on) and their low-level actions (e.g. avoiding getting in each other’s way). In a self-play evaluation, Bayesian Delegation outperforms alternative algorithms. Bayesian Delegation is also a capable ad-hoc collaborator and successfully coordinates with other agent types even in the absence of prior experience.

Quantifying the effects of environment and population diversity in multi-agent reinforcement learning

“…Overcooked (Figure 1c) is a common-payoff game, played in the experiments here by n = 2 players (Carroll et al, 2019; see also Charakorn et al, 2020;Knott et al, 2021;Wang et al, 2020).…”

The paradox of social interaction: Shared intentionality, we-reasoning, and virtual bargaining.

“…One particularly interesting approach along these lines, which can be viewed as formalizing the theory-theory approach, is “action interpretation as inverse planning” (Baker et al, 2009; Wang et al, 2021). The aim is to provide a reason-based account of social perception by applying a Bayesian analysis-by-synthesis approach, widely used in modeling perception (Yuille & Kersten, 2006) and language interpretation (Bever & Poeppel, 2010), to the problem of understanding the intentions behind observed actions.…”

“…15This viewpoint is closely linked with important recent computational work in a Bayesian framework on a somewhat different, and in some ways more challenging problem (Kleiman-Weiner et al, 2016; Wang et al, 2021). In this work, the agents do not have the “payoffs” of possible combinations of actions in common ground, but must learn them from experience; and they must also learn the “utilities” of others, and whether the others are collaborating for the benefit of the team, or are maximizing their own rewards (Kleiman-Weiner et al, 2016).…”

“…In this work, the agents do not have the “payoffs” of possible combinations of actions in common ground, but must learn them from experience; and they must also learn the “utilities” of others, and whether the others are collaborating for the benefit of the team, or are maximizing their own rewards (Kleiman-Weiner et al, 2016). Recently, Wang et al (2021) have proposed an interesting algorithm, Bayesian delegation, in which each agent attempts to infer the tasks of the entire set of agents using Bayesian inference from observing their actions. The payoffs to each agent depend on the interaction of different agents on a task (e.g., collaborating on some subtasks, distributing other subtasks between themselves).…”

The computational society

“…But how are such provisional and flexible agreements reached, often with minimal or no communication? One suggestion is that each party is engaged in a process of simulated bargaining, working out what would be agreed about word meanings, objectives, and plans, were negotiation possibleand generated actions and communicative signals which help coordinate the ongoing negotiation (for this and related approaches, see [2][3][4][5]). More broadly, the idea that social life is governed by layers of such implicit bargains, building upon each other over long periods of cultural evolution, provides a possible mechanism for the emergence of social norms, conventions, and ultimately codified rules and legal systems.…”