Closed-Form Whittle’s Index-Enabled Random Access for Timely Status Update

Sun, Jingzhou; Jiang, Zhiyuan; Krishnamachari, Bhaskar; Zhou, Sheng; Niu, Zhisheng

doi:10.1109/tcomm.2019.2960346

Cited by 95 publications

(59 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Afterwards, a series of works [5]- [12] aimed at characterizing the average AoI and its variations (e.g., Peak Age-of-Information (PAoI) [8]- [10] and Value of Information of Update (VoIU) [11]) for adaptations of the queueing model studied in [4]. Another direction of research [13]- [33] focused on employing AoI as a performance metric for different communication systems that deal with time critical information while having limited resources, e.g., multi-server information-update systems [14], broadcast networks [15]- [17], multi-hop networks [18], cognitive networks [19], unmanned aerial vehicle (UAV)assisted communication systems [20]- [22], IoT networks [2], [23], [24], ultra-reliable low-latency vehicular networks [25], multicast networks [26], decentralized random access schemes [32], and multi-state time-varying networks [33]. Particularly, the objective of this research direction was to characterize optimal policies that minimize average AoI, referred to as ageoptimal polices, by applying different tools from optimization theory.…”

Section: A Related Workmentioning

confidence: 99%

A Reinforcement Learning Framework for Optimizing Age of Information in RF-Powered Communication Systems

Abd-Elmagid

Dhillon

Παππάς

2020

IEEE Trans. Commun.

158

View full text Add to dashboard Cite

In this paper, we study a real-time monitoring system in which multiple source nodes are responsible for sending update packets to a common destination node in order to maintain the freshness of information at the destination. Since it may not always be feasible to replace or recharge batteries in all source nodes, we consider that the nodes are powered through wireless energy transfer (WET) by the destination. For this system setup, we investigate the optimal online sampling policy (referred to as the age-optimal policy) that jointly optimizes WET and scheduling of update packet transmissions with the objective of minimizing the long-term average weighted sum of Age of Information (AoI) values for different physical processes (observed by the source nodes) at the destination node, referred to as the sum-AoI. To solve this optimization problem, we first model this setup as an average cost Markov decision process (MDP) with finite state and action spaces. Due to the extreme curse of dimensionality in the state space of the formulated MDP, classical reinforcement learning algorithms are no longer applicable to our problem even for reasonable-scale settings. Motivated by this, we propose a deep reinforcement learning (DRL) algorithm that can learn the age-optimal policy in a computationally-efficient manner. We further characterize the structural properties of the age-optimal policy analytically, and demonstrate that it has a threshold-based structure with respect to the AoI values for different processes. We extend our analysis to characterize the structural properties of the policy that maximizes average throughput for our system setup, referred to as the throughputoptimal policy. Afterwards, we analytically demonstrate that the structures of the age-optimal and throughput-optimal policies are different. We also numerically demonstrate these structures as well as the impact of system design parameters on the optimal achievable average weighted sum-AoI.

show abstract

Section: A Related Workmentioning

confidence: 99%

A Reinforcement Learning Framework for Optimizing Age of Information in RF-Powered Communication Systems

Abd-Elmagid

Dhillon

Παππάς

2020

IEEE Trans. Commun.

158

View full text Add to dashboard Cite

show abstract

“…In [2], [6], [9], the problem of minimizing AoI in wireless networks is considered. The authors formulate a discrete-time decision problem and provide the development and analysis of four low-complexity scheduling policies: a Greedy policy, a randomized policy, a Max-Weight policy and a Whittle's Index policy.…”

Section: Related Workmentioning

confidence: 99%

“…An AoI lower bound was obtained which is conjectured to be asymptotically tight. In the literature [7], [9]- [11], it is found that a Whittle's index policy is practically and asymptotically optimal with many terminals since it schedules a terminal based on a scaled age, i.e., approximately √ ℎ where and ℎ are success probability and AoI of terminal-respectively, which jointly accounts for timeliness and regularity in an optimal way (the Max-Weight policy [2] is effectively observing the same rule). Although several works proposed optimization techniques, the explicit AoI analysis is relatively scarcely treated.…”

Section: Related Workmentioning

confidence: 99%

Analysis on Asymptotic Optimality of Round-Robin Scheduling for Minimizing Age of Information with HARQ

Jiang

Huang

Zhang

et al. 2021

IEICE Trans. Commun.

Self Cite

View full text Add to dashboard Cite

In a heterogeneous unreliable multiaccess network, wherein terminals share a common wireless channel with distinct error probabilities, existing works have shown that a persistent round-robin (RR-P) scheduling policy can be arbitrarily worse than the optimum in terms of Age of Information (AoI) under standard Automatic Repeat reQuest (ARQ). In this paper, practical Hybrid ARQ (HARQ) schemes which are widelyused in today's wireless networks are considered. We show that RR-P is very close to optimum with asymptotically many terminals in this case, by explicitly deriving tight, closed-form AoI gaps between optimum and achievable AoI by RR-P. In particular, it is rigorously proved that for RR-P, under HARQ models concerning fading channels (resp. finite-blocklength regime), the relative AoI gap compared with the optimum is within a constant of 6.4% (resp. 6.2% with error exponential decay rate of 0.5). In addition, RR-P enjoys the distinctive advantage of implementation simplicity with channel-unaware and easy-to-decentralize operations, making it favorable in practice. A further investigation considering constraint imposed on the number of retransmissions is presented. The performance gap is indicated through numerical simulations.

show abstract

“…Thus, while the MARBP is generally intractable, as it is known to be PSPACE-hard (see [35]), Whittle introduced, in [29], a widely applied heuristic index policy. For a sample of recent applications, see, for example [36][37][38][39][40][41][42][43][44][45][46][47][48]. Yet, the Whittle index is only defined for a limited class of restless bandits, called indexable, and it is nontrivial to verify whether such an indexability property holds for a given model.…”

Section: Approach Via Restless Bandit Reformulation Whittle Index Amentioning

confidence: 99%

Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

Niño‐Mora

2020

Mathematics

View full text Add to dashboard Cite

We consider the multi-armed bandit problem with penalties for switching that include setup delays and costs, extending the former results of the author for the special case with no switching delays. A priority index for projects with setup delays that characterizes, in part, optimal policies was introduced by Asawa and Teneketzis in 1996, yet without giving a means of computing it. We present a fast two-stage index computing method, which computes the continuation index (which applies when the project has been set up) in a first stage and certain extra quantities with cubic (arithmetic-operation) complexity in the number of project states and then computes the switching index (which applies when the project is not set up), in a second stage, with quadratic complexity. The approach is based on new methodological advances on restless bandit indexation, which are introduced and deployed herein, being motivated by the limitations of previous results, exploiting the fact that the aforementioned index is the Whittle index of the project in its restless reformulation. A numerical study demonstrates substantial runtime speed-ups of the new two-stage index algorithm versus a general one-stage Whittle index algorithm. The study further gives evidence that, in a multi-project setting, the index policy is consistently nearly optimal.

show abstract

Closed-Form Whittle’s Index-Enabled Random Access for Timely Status Update

Cited by 95 publications

References 32 publications

A Reinforcement Learning Framework for Optimizing Age of Information in RF-Powered Communication Systems

A Reinforcement Learning Framework for Optimizing Age of Information in RF-Powered Communication Systems

Analysis on Asymptotic Optimality of Round-Robin Scheduling for Minimizing Age of Information with HARQ

Fast Two-Stage Computation of an Index Policy for Multi-Armed Bandits with Setup Delays

Contact Info

Product

Resources

About