Dynamic Charging Scheme Problem With Actor–Critic Reinforcement Learning

Yang, Meiyi; Liu, Nianbo; Zuo, Lin; Feng, Yong; Liu, Minghui; Gong, Haigang; Liu, Ming

doi:10.1109/jiot.2020.3005598

Cited by 33 publications

(22 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To enhance the network utility, Zhao et al optimized the charging vehicle route, the data rate, and the charging time together, and they presented multiple period iterative algorithms [16]. Considering that the energy consumption rate of nodes is dynamically changed, Yang et al proposed a real-time global charging scheme in wireless rechargeable sensor networks based on the actor-critic reinforcement learning algorithm [17]. Although existing studies have made many achievements from the perspectives of charging time, data transmission delay, network utility, etc., more work is still needed, as the mobility and communication performance of mobile charging vehicles are affected by terrain and other factors, while UAV assisted charging is rarely limited.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Petri-Net Based Multi-Objective Optimization in Multi-UAV Aided Large-Scale Wireless Power and Information Transfer Networks

Qin

Zhao

et al. 2021

Remote Sensing

View full text Add to dashboard Cite

Power consumption in wireless sensor networks is high, and the lifetime of a battery has become a bottleneck, restricting network performance. Wireless power transfer with a ground mobile charger is vulnerable to interference from the terrain and other factors, and hence it is difficult to deploy in practice. Accordingly, a novel paradigm is adopted where a multi-UAV (unmanned aerial vehicle) with batteries can transfer power and information to SDs (sensor devices) in a large-scale sensor network. However, there are discrete events, continuous process, time delay, and decisions in such a complicated system. From the perspective of a hybrid system, a hybrid colored cyber Petri net system is proposed here to depict and analyze this problem. Furthermore, the energy utilization rate and information collection time delay are conflict with each other; therefore, UAV-aided wireless power and information transfer is formulated as a multi-objective optimization problem. For this reason, the MAC-NSGA II (multiple ant colony-nondominated sorting genetic algorithm II) is proposed in this work. Firstly, the optimal trajectory of multiple UAVs was obtained, and on this basis, the above two objectives were optimized simultaneously. Large-scale simulation results show that the proposed algorithm is superior to NSGA II and MOEA/D in terms of energy efficiency and information collection delay.

show abstract

Section: Related Workmentioning

confidence: 99%

“…However, the drawbacks are also obvious. The path planning and communication quality of ground mobile chargers are susceptible to the terrain, and therefore it is difficult to replenish energy efficiently and to do so in a timely way [11][12][13][14][15][16][17].…”

Section: Introductionmentioning

confidence: 99%

Petri-Net Based Multi-Objective Optimization in Multi-UAV Aided Large-Scale Wireless Power and Information Transfer Networks

Qin

Zhao

et al. 2021

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…However, these methods have very slow convergence [35]. To tackle this problem, the Deep Deterministic Policy Gradient abbreviated as DDPG is utilized to integrate both properties of policy-based and valuebased algorithms in order to deal with continuous and large state/action spaces [35], [36]. In this algorithm, there are two separated neural networks, an actor and a critic network with parameters ω and θ, respectively.…”

Section: ) Deep Deterministic Policy Gradient Deep Rl Algorithmmentioning

confidence: 99%

Age of Information Aware VNF Scheduling in Industrial IoT Using Deep Reinforcement Learning

Akbari¹,

Abedi²,

Joda³

et al. 2021

Preprint

View full text Add to dashboard Cite

In delay-sensitive industrial internet of things (IIoT) applications, the age of information (AoI) is employed to characterize the freshness of information. Meanwhile, the emerging network function virtualization provides flexibility and agility for service providers to deliver a given network service using a sequence of virtual network functions (VNFs). However, suitable VNF placement and scheduling in these schemes is NP-hard and finding a globally optimal solution by traditional approaches is complex. Recently, deep reinforcement learning (DRL) has appeared as a viable way to solve such problems. In this paper, we first utilize single agent low-complex compound action actor-critic RL to cover both discrete and continuous actions and jointly minimize VNF cost and AoI in terms of network resources under end-toend Quality of Service constraints. To surmount the single-agent capacity limitation for learning, we then extend our solution to a multi-agent DRL scheme in which agents collaborate with each other. Simulation results demonstrate that single-agent schemes significantly outperform the greedy algorithm in terms of average network cost and AoI. Moreover, multi-agent solution decreases the average cost by dividing the tasks between the agents. However, it needs more iterations to be learned due to the requirement on the agents' collaboration.

show abstract

“…Cao et al [ 28 ] proposed a deep reinforcement learning-based on-demand charging algorithm to maximize the sum of rewards collected by the mobile charger in WRSN, which is subject to the energy capacity constraint on the mobile charger and the charging times of all sensor nodes. A novel charging scheme for dynamic WRSNs based on an actor–critic reinforcement learning algorithm was proposed by Yang et al [ 31 ], which aimed to maximize the charging efficiency while minimizing the number of dead sensors to prolong the network lifetime. The above works have made significant model innovation and algorithm innovation, yet they ignore the impact of sensor charging energy on the optimization performance.…”

Section: Introductionmentioning

confidence: 99%

“…The above works have made significant model innovation and algorithm innovation, yet they ignore the impact of sensor charging energy on the optimization performance. Although Yang et al [ 31 ] proposed a charging coefficient to constrain the upper charging energy threshold, they assumed that all sensors have a fixed charging coefficient during the scheduling, which cannot adjust according to the needs of the sensors. Specifically, the charging coefficient could directly determine the charging energy for the sensor.…”

Section: Introductionmentioning

confidence: 99%

Attention-Shared Multi-Agent Actor–Critic-Based Deep Reinforcement Learning Approach for Mobile Charging Dynamic Scheduling in Wireless Rechargeable Sensor Networks

Jiang

Wang

Chen

et al. 2022

Entropy

View full text Add to dashboard Cite

The breakthrough of wireless energy transmission (WET) technology has greatly promoted the wireless rechargeable sensor networks (WRSNs). A promising method to overcome the energy constraint problem in WRSNs is mobile charging by employing a mobile charger to charge sensors via WET. Recently, more and more studies have been conducted for mobile charging scheduling under dynamic charging environments, ignoring the consideration of the joint charging sequence scheduling and charging ratio control (JSSRC) optimal design. This paper will propose a novel attention-shared multi-agent actor–critic-based deep reinforcement learning approach for JSSRC (AMADRL-JSSRC). In AMADRL-JSSRC, we employ two heterogeneous agents named charging sequence scheduler and charging ratio controller with an independent actor network and critic network. Meanwhile, we design the reward function for them, respectively, by considering the tour length and the number of dead sensors. The AMADRL-JSSRC trains decentralized policies in multi-agent environments, using a centralized computing critic network to share an attention mechanism, and it selects relevant policy information for each agent at every charging decision. Simulation results demonstrate that the proposed AMADRL-JSSRC can efficiently prolong the lifetime of the network and reduce the number of death sensors compared with the baseline algorithms.

show abstract

Dynamic Charging Scheme Problem With Actor–Critic Reinforcement Learning

Cited by 33 publications

References 42 publications

Petri-Net Based Multi-Objective Optimization in Multi-UAV Aided Large-Scale Wireless Power and Information Transfer Networks

Petri-Net Based Multi-Objective Optimization in Multi-UAV Aided Large-Scale Wireless Power and Information Transfer Networks

Age of Information Aware VNF Scheduling in Industrial IoT Using Deep Reinforcement Learning

Attention-Shared Multi-Agent Actor–Critic-Based Deep Reinforcement Learning Approach for Mobile Charging Dynamic Scheduling in Wireless Rechargeable Sensor Networks

Contact Info

Product

Resources

About