Policy search for multi-robot coordination under uncertainty

Abstract

We introduce a principled method for multi-robot coordination based on a general model (termed a MacDec-POMDP) of multi-robot cooperative planning in the presence of stochasticity, uncertain sensing, and communication limitations. A new MacDec-POMDP planning algorithm is presented that searches over policies represented as finite-state controllers, rather than the previous policy tree representation. Finite-state controllers can be much more concise than trees, are much easier to interpret, and can operate over an infinite horizon. The resulting policy search algorithm requires a substantially simpler simulator that models only the outcomes of executing a given set of motor controllers, not the details of the executions themselves and can solve significantly larger problems than existing MacDec-POMDP planners. We demonstrate significant performance improvements over previous methods and show that our method can be used for actual multi-robot systems through experiments on a cooperative multi-robot bartending domain.

Decision-Making Under Uncertainty in Multi-Agent and Multi-Robot Systems: Planning and Learning

“…Because tree-based representations become intractable as the horizon grows, we also developed multiple methods for optimizing finite-state controllers in macro-action Dec-POMDPs using the MacDec-POMDP [Amato et al, 2015a] and Dec-POSMDP [Omidshafiei et al, 2015] models. Some of these approaches can provide solutions with only a highlevel model of the macro-actions (i.e., distributions over time and outcomes) instead of a full model of the underlying Dec-POMDP [Amato et al, 2015a;2017;2017a]. Another approach automatically generates the macro-actions from low-level (continuous) dynamics models , while another method generates (macro-)observations from low-level sensor (e.g., camera) information [Omidshafiei et al, 2017b].…”

“…We call this model a MacDec-POMDP [Amato et al, 2014] when the low-level Dec-POMDP model and the policies of the macro-actions are known and a decentralized partially observable semi-Markov decision process (Dec-POSMDP) when a high-level model is defined which includes time to completion Amato et al, 2015a] (but a simulator can be used in place of a model in each case). While these high-level models still include the states of the Dec-POMDP, they do not include the Dec-POMDP actions and observations.…”

“…By extending Dec-POMDP algorithms to the macro-action case, realistic multi-agent coordination problems can be solved that are orders of magnitude larger than problems solved by previous methods and continuous state spaces can be considered, including for multiple mobile robots in warehouse [Amato et al, 2015b], logistics [Amato et al, 2015a;2017] and aerial delivery [Omidshafiei et al, 2015; 2017a; 2017b] scenarios. We discuss some of these methods and domains below.…”

“…We demonstrated our macro-action-based approaches on multiple multi-robot domains with limited sensing and communication. These domains included a warehousing problem, where robots have uncertainty about the location of boxes and limited communication, but must coordinate to collect the large boxes and individually push the small boxes to the shipping location [Amato et al, 2015b]; a logistics (beer delivery) domain, where two robots must efficiently find out about and service beer orders in cooperation with a 'picker/bartender' robot, which can retrieve items [Amato et al, 2015a;2017]; and a package delivery domain, where a group of aerial robots must retrieve and deliver packages from base locations to delivery locations while dealing with limited battery life 2016;2017b;2017a]. 1 Our methods outperformed simpler methods and 'expert' hand-coded solutions (e.g., delivering an additional three beers every 10 minutes).…”

An Evaluation of Multi-Robot Systems Exploration Algorithms

“…Exploring algorithm determines the shortest and the lowest-cost path of a robot based on the environment size and type, cell size, and the numbers of robots constitute the exploration team and update the environment map based on the gathered information [3,24]. A set of common exploration algorithms will be discussed in section 4.…”