Learning to Collaborate in Markov Decision Processes

Abstract

We consider a two-agent MDP framework where agents repeatedly solve a task in a collaborative setting. We study the problem of designing a learning algorithm for the first agent (A 1 ) that facilitates successful collaboration even in cases when the second agent (A 2 ) is adapting its policy in an unknown way. The key challenge in our setting is that the first agent faces non-stationarity in rewards and transitions because of the adaptive behavior of the second agent.We design novel online learning algorithms for agent A 1 whose regret decays as O T max{1− 3 7 •α, 1 4 } , for T learning episodes, provided that the magnitude in the change in agent A 2 's policy between any two consecutive episodes is upper bounded by O(T −α ). Here, the parameter α is assumed to be strictly greater than 0, and we show that this assumption is necessary provided that the learning parity with noise problem is computationally hard. We show that sublinear regret of agent A 1 further implies nearoptimality of the agents' joint return for MDPs that manifest the properties of a smooth game.

Dynamic Regret of Policy Optimization in Non-stationary Environments

“…Assumption 1 states that the visitation measures do not change drastically when similar policies are executed. This notion of smoothness in visitation measures also appears in [41] in the context of two-player games.…”

“…In addition to updating and evaluating policy, Algorithm 1 features a periodic restart mechanism, which resets its policy estimate every τ episodes. Restart mechanisms have been used to handle non-stationarity in RL [27,39] and related problems including bandits [6], online convex optimization [7,26] and games [17,41]. Intuitively, by employing the restart mechanism, Algorithm 1 is able to stabilize its iterates against non-stationary drift in the learning process due to adversarial reward functions.…”

“…In particular, the reward mechanism may vary across tasks, and therefore requires the agent to adapt to the change of tasks. Another example of RL under non-stationary environments is humanmachine interaction [23,41]. This line of research studies how humans and machines (or robots) should interact or collaborate to accomplish certain goals.…”

Provable Fictitious Play for General Mean-Field Games

“…Finally, our last assumption stipulates that the state visitation distributions are smooth with respect to the (embedded) mean-field states of the MFG. This assumption is analogous to those in the literature on MDP and two-player games (Fei et al, 2020;Radanovic et al, 2019), which requires the visitation distributions to be smooth with respect to the policy. Assumption 5.…”