Reinforcement Learning for Minimizing Age of Information under Realistic Physical Dynamics

Wang, Sihua; Chen, Mingzhe; Saad, Walid; Yin, Changchuan; Cui, Shuguang; Poor, H. Vincent

doi:10.1109/globecom42002.2020.9322139

Cited by 7 publications

(4 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1) Sampling-Communication Co-design: AoI is a performance metric widely used in co-design communication systems and sampling policies (also called state update policies) [13]- [15]. In [13], the authors optimized the sensing and updating policy for an air pollution monitoring application by minimizing the weighted sum of the AoI and the total energy consumption of the device. By adjusting the weighting coefficients of the AoI and tuning the energy consumption manually, it is possible to achieve the target trade-off.…”

Section: A Related Workmentioning

confidence: 99%

Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse

Meng

She

Zhao

et al. 2023

IEEE J. Select. Areas Commun.

View full text Add to dashboard Cite

The metaverse has the potential to revolutionize the next generation of the Internet by supporting highly interactive services with the help of Mixed Reality (MR) technologies; still, to provide a satisfactory experience for users, the synchronization between the physical world and its digital models is crucial. This work proposes a sampling, communication and prediction codesign framework to minimize the communication load subject to a constraint on tracking the Mean Squared Error (MSE) between a real-world device and its digital model in the metaverse. To optimize the sampling rate and the prediction horizon, we exploit expert knowledge and develop a constrained Deep Reinforcement Learning (DRL) algorithm, named Knowledgeassisted Constrained Twin-Delayed Deep Deterministic (KC-TD3) policy gradient algorithm. We validate our framework on a prototype composed of a real-world robotic arm and its digital model. Compared with existing approaches: (1) When the tracking error constraint is stringent (MSE = 0.002 • ), our policy degenerates into the policy in the sampling-communication co-design framework. (2) When the tracking error constraint is mild (MSE = 0.007 • ), our policy degenerates into the policy in the prediction-communication co-design framework. (3) Our framework achieves a better trade-off between the average MSE and the average communication load compared with a communication system without sampling and prediction. For example, the average communication load can be reduced up to 87% when the average track error constraint is 0.002 • . (4) Our policy outperforms the benchmark with the static sampling rate and prediction horizon optimized by exhaustive search, in terms of the tail probability of the tracking error. Furthermore, with the assistance of expert knowledge, the proposed algorithm KC-TD3 achieves better convergence time, stability, and final policy performance.

show abstract

Section: A Related Workmentioning

confidence: 99%

Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse

Meng

She

Zhao

et al. 2023

IEEE J. Select. Areas Commun.

View full text Add to dashboard Cite

show abstract

“…In earlier works, the authors in [35] presented RL-based algorithm for optimal UAV positioning and transmission of power to drone small cells in order to revamp the outage performance and energy efficiency of UAVs. In [36], a UAVassisted wireless sensor network is considered and authors developed a distributed RL strategy that permits devices to collaboratively update RL parameters. The objective was to minimize the weighted sum of Age of Information (AoI) cost in real-time and total energy consumption.…”

Section: A Related Workmentioning

confidence: 99%

A DRL Strategy for Optimal Resource Allocation Along With 3D Trajectory Dynamics in UAV-MEC Network

et al. 2023

View full text Add to dashboard Cite

Advances in Unmanned Air Vehicle (UAV) technology have paved a way for numerous configurations and applications in communication systems. However, UAV dynamics play an important role in determining its effective use. In this article, while considering UAV dynamics, we evaluate the performance of a UAV equipped with a Mobile-Edge Computing (MEC) server that provides services to End-user Devices (EuDs). The EuDs due to their limited energy resources offload a portion of their computational task to nearby MEC-based UAV. To this end, we jointly optimize the computational cost and 3D UAV placement along with resource allocation subject to the network, communication, and environment constraints. A Deep Reinforcement Learning (DRL) technique based on a continuous action space approach, namely Deep Deterministic Policy Gradient (DDPG) is utilized. By exploiting DDPG, we propose an optimization strategy to obtain an optimal offloading policy in the presence of UAV dynamics, which is not considered in earlier studies. The proposed strategy can be classified into three cases namely; training through an ideal scenario, training through error dynamics, and training through extreme values. We compared the performance of these individual cases based on cost percentage and concluded that case II (training through error dynamics) achieves minimum cost i.e., 37.75 %, whereas case I and case III settles at 67.25% and 67.50% respectively. Numerical simulations are performed, and extensive results are obtained which shows that the advanced DDPG based algorithm along with error dynamic protocol is able to converge to near optimum. To validate the efficacy of the proposed algorithm, a comparison with state-of-the-art Deep Q-Network (DQN) is carried out, which shows that our algorithm has significant improvements.

show abstract

“…Chen et al [13] investigated the AoI-aware radio resource management problem for an expected long-term performance optimization in a Manhattan grid V2V communication network and proposed a proactive algorithm based on the deep recurrent Qnetwork. Wu et al [23] considered cellular Internet assisted by UAVs, studied the UAV's AoI minimization problem by designing the UAV's trajectory. Wang et al [24] studied the problem of minimizing the weighted sum of the AoI and the total energy consumption of IoT devices.…”

Section: Age Of Informationmentioning

confidence: 99%

Towards V2I Age-Aware Fairness Access: A DQN Based Intelligent Vehicular Node Training and Test Method

Wu¹,

Shi²,

Wan³

et al. 2023

Chinese J. Elect.

View full text Add to dashboard Cite

Vehicles on the road exchange data with base station frequently through vehicle to infrastructure (V2I) communications to ensure the normal use of vehicular applications, where the IEEE 802.11 distributed coordination function is employed to allocate a minimum contention window (MCW) for channel access. Each vehicle may change its MCW to achieve more access opportunities at the expense of others, which results in unfair communication performance. Moreover, the key access parameter MCW is privacy information and each vehicle is not willing to share it with other vehicles. In this uncertain setting, age of information (AoI), which measures the freshness of data and is closely related with fairness, has become an important communication metric. On this basis, we design an intelligent vehicular node to learn the dynamic environment and predict the optimal MCW, which can make the intelligent node achieve age fairness. In order to allocate the optimal MCW for the vehicular node, we employ a learning algorithm to make a desirable decision by learning from replay history data. In particular, the algorithm is proposed by extending the traditional deep-Q-learning (DQN) training and testing method. Finally, by comparing with other methods, it is proved that the proposed DQN method can significantly improve the age fairness of the intelligent node.

show abstract

Reinforcement Learning for Minimizing Age of Information under Realistic Physical Dynamics

Cited by 7 publications

References 29 publications

Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse

Sampling, Communication, and Prediction Co-Design for Synchronizing the Real-World Device and Digital Model in Metaverse

A DRL Strategy for Optimal Resource Allocation Along With 3D Trajectory Dynamics in UAV-MEC Network

Towards V2I Age-Aware Fairness Access: A DQN Based Intelligent Vehicular Node Training and Test Method

Contact Info

Product

Resources

About