RETRACTED: Hierarchical RNNs-Based transformers MADDPG for mixed cooperative-competitive environments

Abstract

Structural models based on Attention can not only record the relationships between features’ position, but also can measure the importance of different features based on their weights. By establishing dynamically weighted parameters for choosing relevant and irrelevant features, the key information can be strengthened, and the irrelevant information can be weakened. Therefore, the efficiency of Deep Learning algorithms can be significantly elevated and improved. Although Transformer have been performed very well in many fields including Reinforcement Learning (RL). We tried to integrate Transformers into RL, however there are some challenge in this task. Especially, MARL (known as Multi-Agent Reinforcement Learning), which can be recognized as a set of independent agents trying to adapt and learn through their way to reach the goal. In order to emphasize the relationship between each MDP decision in a certain time period, we applied the hierarchical coding method and validated the effectiveness of this method. This paper proposed a Hierarchical Transformer MADDPG based on recurrent neural network(RNN) which we call it Hierarchical RNNs-Based Transformers MADDPG(HRTMADDPG). It consists of a lower level encoder based on RNNs that encodes multiple step sizes in each time sequence, and it also consists of an upper sequence level encoder based on Transformer for learning the correlations between multiple sequences. Then we can capture the causal relationship between sub-time sequences and make HRTMADDPG more efficient.

Graph MADDPG with RNN for multiagent cooperative environment

“…More precisely, a transformer consists of self-attention and a feedforward neural network. In the studies mentioned in references (Iqbal and Sha, 2019 ; Wei et al, 2021 ), many contributions to MADDPG and the attention mechanism were made, which achieved remarkable results.…”

“…Our previous results (Wei et al, 2021 ) proposed a hierarchical transformer MADDPG based on the RNN method, which inherited the idea of the RNN and added transformer at the same time. To verify that the graph convolution with a transformer can indeed express more features of the agent, we compared the reward values of the three sets in the experiments.…”

“…Iqbal and Sha ( 2019 ) proposed a method by combining attention and MADDPG. Significant approaches (Jiang and Lu, 2018 ; Yathartha and Enna Sachdeva, 2019 ; Wei et al, 2022 ) have made great improvements in MADDPG. The MADDPG (Mordatch and Abbeel, 2018 ) environment is based on time series.…”

Decentralized power grid control scheme utilizing photonic sensing and computing

“…Promising frameworks have been proposed by Open AI's MADDPG 13 and even hierarchical RNNs-Based Transformers (the most complex in terms of neural network architecture). 24 The primary reason for doing this is because grid operation has typically concerned one objective (cost), and when multiple objectives are defined, or multi-level optimization is used, it concerns future planning strategy or emission reduction. 25 As mentioned before, even if these complex optimization formulations encompass more objectives, there is a price to pay when solving the problem.…”

“…The next step is to try the most complex emerging architectures that deal with cooperative/competitive games, such as hierarchical recurrent transformer-based MADDPG (HRTMADDPG). 24 As we adopt these techniques and apply them to grid control problems, we start to see a new kind of multi-layered, multi-valued, coordination depicted in Figure 6. And because of the accelerated sensing and computing, we can explore these boundaries.…”