Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces

Fu, Haotian; Tang, Hongyao; Hao, Jianye; Lei, Zihan; Chen, Yingfeng; Fan, Changjie

doi:10.24963/ijcai.2019/323

Cited by 41 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Those methods assume on-policy and handle discrete and continuous actions separately. Xiong et al [36] and Fu et al [37] use a hierarchical structure [38] to deal with the discrete actions and generate the continuous action based on the discrete actions. Neunert et al [39] provided mixed policy to handle the discrete-continuous action space.…”

Section: Related Workmentioning

confidence: 99%

D3PG: Dirichlet DDGP for Task Partitioning and Offloading with Constrained Hybrid Action Space in Mobile Edge Computing

Ale¹,

King²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

<div> Mobile Edge Computing (MEC) has been regarded as a promising paradigm to reduce service latency for data processing in Internet of Things, by provisioning computing resources at network edge. In this work, we jointly optimize the task partitioning and computational power allocation for computation offloading in a dynamic environment with multiple IoT devices and multiple edge servers. We formulate the problem as a Markov decision process with constrained hybrid action space, which cannot be well handled by existing deep reinforcement learning (DRL) algorithms. Therefore, we develop a novel Deep Reinforcement Learning called Dirichlet Deep Deterministic Policy Gradient (D3PG), which </div><div>is built on Deep Deterministic Policy Gradient (DDPG) to solve the problem. The developed model can learn to solve multi-objective optimization, including maximizing the number of tasks processed before expiration and minimizing the energy cost and service latency. More importantly, D3PG can effectively deal with constrained distribution-continuous hybrid action space, where the distribution variables are for the task partitioning and offloading, while the continuous variables are for computational frequency control. Moreover, the D3PG can address many similar issues in MEC and general reinforcement learning problems. Extensive simulation results show that the proposed D3PG outperforms the state-of-art methods.</div><div> Mobile Edge Computing (MEC) has been regarded as a promising paradigm to reduce service latency for data processing in Internet of Things, by provisioning computing resources at network edge. In this work, we jointly optimize the task partitioning and computational power allocation for computation offloading in a dynamic environment with multiple IoT devices and multiple edge servers. We formulate the problem as a Markov decision process with constrained hybrid action space, which cannot be well handled by existing deep reinforcement learning (DRL) algorithms. Therefore, we develop a novel Deep Reinforcement Learning called Dirichlet Deep Deterministic Policy Gradient (D3PG), which is built on Deep Deterministic Policy Gradient (DDPG) to solve the problem. The developed model can learn to solve multi-objective optimization, including maximizing the number of tasks processed before expiration and minimizing the energy cost and service latency. More importantly, D3PG can effectively deal with constrained distribution-continuous hybrid action space, where the distribution variables are for the task partitioning and offloading, while the continuous variables are for computational frequency control. Moreover, the D3PG can address many similar issues in MEC and general reinforcement learning problems. Extensive simulation results show that the proposed D3PG outperforms the state-of-art methods.</div>

show abstract

Section: Related Workmentioning

confidence: 99%

D3PG: Dirichlet DDGP for Task Partitioning and Offloading with Constrained Hybrid Action Space in Mobile Edge Computing

Ale¹,

King²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…As we have discussed in the above, P-DQN algorithm [41] is applicable to the hybrid action space of a single agent. This algorithm is then extended to the environment with multiple cooperative agents [42]. However, traders in the double auction market are not cooperative, and therefore, we extend it to the environment with multiple noncooperative agents, called I-PDQN algorithm.…”

Section: Nash Equilibrium Trading Strategymentioning

confidence: 99%

Trading and Pricing Sensor Data in Competing Edge Servers with Double Auction Markets

Shi

Song

2021

Journal of Sensors

View full text Add to dashboard Cite

With the development of the IoT (Internet of Things), sensors networks can bring a large amount of valuable data. In addition to be utilized in the local IoT applications, the data can also be traded in the connected edge servers. As an efficient resource allocation mechanism, the double auction has been widely used in the stock and futures markets and can be also applied in the data resource allocation in sensor networks. Currently, there usually exist multiple edge servers running double auctions competing with each other to attract data users (buyers) and producers (sellers). Therefore, the double auction market run on each edge server needs efficient mechanism to improve the allocation efficiency. Specifically, the pricing strategy of the double auction plays an important role on affecting traders’ profit, and thus, will affect the traders’ market choices and bidding strategies, which in turn affect the competition result of double auction markets. In addition, the traders’ trading strategies will also affect the market’s pricing strategy. Therefore, we need to analyze the double auction markets’ pricing strategy and traders’ trading strategies. Specifically, we use a deep reinforcement learning algorithm combined with mean field theory to solve this problem with a huge state and action space. For trading strategies, we use the Independent Parametrized Deep Q-Network (I-PDQN) algorithm combined with mean field theory to compute the Nash equilibrium strategies. We then compare it with the fictitious play (FP) algorithm. The experimental results show that the computation speed of I-PDQN algorithm is significantly faster than that of FP algorithm. For pricing strategies, the double auction markets will dynamically adjust the pricing strategy according to traders’ trading strategies. This is a sequential decision-making process involving multiple agents. Therefore, we model it as a Markov game. We adopt Multiagent Deep Deterministic Policy Gradient (MADDPG) algorithm to analyze the Nash equilibrium pricing strategies. The experimental results show that the MADDPG algorithm solves the problem faster than the FP algorithm.

show abstract

“…Deep learning is a major advancement in the field of machine learning in recent years, which has aroused great interest of researchers and is widely used in several machine learning tasks, including computer vision, image analysis, speech recognition, information retrieval, natural language processing, reinforcement learning and multi-agent systems [16][17][18][19][20][21][22][23]. In addition, in the field of bioinformatics, deep learning is also widely used.…”

Section: Graph Convolutional Networkmentioning

confidence: 99%

Combining sequence and network information to enhance protein–protein interaction prediction

Liu

Zhu

et al. 2020

BMC Bioinformatics

View full text Add to dashboard Cite

Background Protein–protein interactions (PPIs) are of great importance in cellular systems of organisms, since they are the basis of cellular structure and function and many essential cellular processes are related to that. Most proteins perform their functions by interacting with other proteins, so predicting PPIs accurately is crucial for understanding cell physiology. Results Recently, graph convolutional networks (GCNs) have been proposed to capture the graph structure information and generate representations for nodes in the graph. In our paper, we use GCNs to learn the position information of proteins in the PPIs networks graph, which can reflect the properties of proteins to some extent. Combining amino acid sequence information and position information makes a stronger representation for protein, which improves the accuracy of PPIs prediction. Conclusion In previous research methods, most of them only used protein amino acid sequence as input information to make predictions, without considering the structural information of PPIs networks graph. We first time combine amino acid sequence information and position information to make representations for proteins. The experimental results indicate that our method has strong competitiveness compared with several sequence-based methods.

show abstract

Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces

Cited by 41 publications

References 0 publications

D3PG: Dirichlet DDGP for Task Partitioning and Offloading with Constrained Hybrid Action Space in Mobile Edge Computing

D3PG: Dirichlet DDGP for Task Partitioning and Offloading with Constrained Hybrid Action Space in Mobile Edge Computing

Trading and Pricing Sensor Data in Competing Edge Servers with Double Auction Markets

Combining sequence and network information to enhance protein–protein interaction prediction

Contact Info

Product

Resources

About