Efficient policy detecting and reusing for non-stationarity in Markov games

Yan, Zheng; Hao, Jianye; Zhang, Zongzhang; Meng, Zhaopeng; Yang, Tianpei; Li, Yanran; Cheng, Fan

doi:10.1007/s10458-020-09480-9

Cited by 13 publications

(5 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As π 1 i and π 2 i are jointly trained to maximize performance, the optimal response to π 2 i is already contained within π 1 i . Thus, inspired by recent success in distillation (Zheng et al 2021), we treat π 1 i as an expert and let π M distill its policy. During sampling, we collect across-episodic trajectories, denoted as h t = (o 0 , a 0 , r 0 , ..., o t , a t , r t ).…”

Section: Approachmentioning

confidence: 99%

Multi-Expert Distillation for Few-Shot Coordination (Student Abstract)

Zhu,

Ding,

Zhang

2024

AAAI

View full text Add to dashboard Cite

Ad hoc teamwork is a crucial challenge that aims to design an agent capable of effective collaboration with teammates employing diverse strategies without prior coordination. However, current Population-Based Training (PBT) approaches train the ad hoc agent through interaction with diverse teammates from scratch, which suffer from low efficiency. We introduce Multi-Expert Distillation (MED), a novel approach that directly distills diverse strategies through modeling across-episodic sequences. Experiments show that our algorithm achieves more efficient and stable training and has the ability to improve its behavior using historical contexts. Our code is available at https://github.com/LAMDA-RL/MED.

show abstract

Section: Approachmentioning

confidence: 99%

Multi-Expert Distillation for Few-Shot Coordination (Student Abstract)

Zhu,

Ding,

Zhang

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Instead of applying BPR to detect opponents' policies, another approach named Bayes-Theory of Mind on Policy (Bayes-ToMoP) is proposed in [152] to efficiently detect the strategy of opponents using either stationary or higher-level reasoning strategies. By combining (BPR+) with the distill policy network (DPN), the DPN-BPR+ is proposed in [153] to study studies efficient policy detecting and reusing techniques in nonstationary MGs. Nonstationarity also prevents the straightforward use of experience replay, which is crucial for stabilizing RL.…”

Section: B Microagent Behavioral Interventionmentioning

confidence: 99%

Synergistic Integration Between Machine Learning and Agent-Based Modeling: A Multidisciplinary Review

Zhang

Valencia

Chang

2023

IEEE Trans. Neural Netw. Learning Syst.

View full text Add to dashboard Cite

Agent-based modeling (ABM) involves developing models in which agents make adaptive decisions in a changing environment. Machine-learning (ML) based inference models can improve sequential decision-making by learning agents' behavioral patterns. With the aid of ML, this emerging area can extend traditional agent-based schemes that hardcode agents' behavioral rules into an adaptive model. Even though there are plenty of studies that apply ML in ABMs, the generalized applicable scenarios, frameworks, and procedures for implementations are not well addressed. In this article, we provide a comprehensive review of applying ML in ABM based on four major scenarios, i.e., microagent-level situational awareness learning, microagent-level behavior intervention, macro-ABM-level emulator, and sequential decision-making. For these four scenarios, the related algorithms, frameworks, procedures of implementations, and multidisciplinary applications are thoroughly investigated. We also discuss how ML can improve prediction in ABMs by trading off the variance and bias and how ML can improve the sequential decision-making of microagent and macrolevel policymakers via a mechanism of reinforced behavioral intervention. At the end of this article, future perspectives of applying ML in ABMs are discussed with respect to data acquisition and quality issues, the possible solution of solving the convergence problem of reinforcement learning, interpretable ML applications, and bounded rationality of ABM.

show abstract

“…A set of assumptions that can be made is that all players have fixed strategy sets. Under these assumptions, agents could maintain more sophisticated beliefs about their opponent (Zheng et al, 2018) and extend this to recursive-reasoning procedures (Yang et al, 2019). These lines of work focus more on other-player policy identification and are a promising future direction for improving the quality of the OPC.…”

Section: Related Work Multiagent Learningmentioning

confidence: 99%

Learning to play against any mixture of opponents

Smith

Anthony²,

Wang³

et al. 2023

Front. Artif. Intell.

View full text Add to dashboard Cite

Intuitively, experience playing against one mixture of opponents in a given domain should be relevant for a different mixture in the same domain. If the mixture changes, ideally we would not have to train from scratch, but rather could transfer what we have learned to construct a policy to play against the new mixture. We propose a transfer learning method, Q-Mixing, that starts by learning Q-values against each pure-strategy opponent. Then a Q-value for any distribution of opponent strategies is approximated by appropriately averaging the separately learned Q-values. From these components, we construct policies against all opponent mixtures without any further training. We empirically validate Q-Mixing in two environments: a simple grid-world soccer environment, and a social dilemma game. Our experiments find that Q-Mixing can successfully transfer knowledge across any mixture of opponents. Next, we consider the use of observations during play to update the believed distribution of opponents. We introduce an opponent policy classifier—trained reusing Q-learning data—and use the classifier results to refine the mixing of Q-values. Q-Mixing augmented with the opponent policy classifier performs better, with higher variance, than training directly against a mixed-strategy opponent.

show abstract

Efficient policy detecting and reusing for non-stationarity in Markov games

Cited by 13 publications

References 38 publications

Multi-Expert Distillation for Few-Shot Coordination (Student Abstract)

Multi-Expert Distillation for Few-Shot Coordination (Student Abstract)

Synergistic Integration Between Machine Learning and Agent-Based Modeling: A Multidisciplinary Review

Learning to play against any mixture of opponents

Contact Info

Product

Resources

About