A Deep Reinforcement Learning-Based Dynamic Computational Offloading Method for Cloud Robotics

Abstract

Robots come with a variety of computing capabilities, and running computationallyintense applications on robots is sometimes challenging on account of limited onboard computing, storage, and power capabilities. Meanwhile, cloud computing provides on-demand computing capabilities, and thus combining robots with cloud computing can overcome the resource constraints robots face. The key to effectively offloading tasks is an application solution that does not underutilize the robot's own computational capabilities and makes decisions based on crucial cost parameters such as latency and CPU availability. In this paper, we formulate the application offloading problem as a Markovian decision process and propose a deep reinforcement learning-based deep Q-network (DQN) approach. The statespace is formulated with the assumption that input data size directly impacts application execution time. The proposed algorithm is designed as a continuous task problem with discrete action space; i.e., we apply a choice of action at each time step and use the corresponding outcome to train the DQN to acquire the maximum rewards possible. To validate the proposed algorithm, we designed and implemented a robot navigation testbed. The results demonstrated that for the given state-space values, the proposed algorithm learned to take appropriate actions to reduce application latency and also learned a policy that takes actions based on input data size. Finally, we compared the proposed DQN algorithm with a long short-term memory (LSTM) algorithm in terms of accuracy. When trained and validated on the same dataset, the proposed DQN algorithm obtained at least 9 percentage points greater accuracy than the LSTM algorithm.INDEX TERMS Cloud robotics, deep reinforcement learning, deep Q-networks (DQN), AWS, neural networks, application offloading, robot navigation.

A smart collaborative framework for dynamic multi-task offloading in IIoT-MEC networks

“…The authors of [22], [23] stress the imperative of avoiding under-utilizing a robot's onboard computational resources for efficient offloading. To address this, they formulate the application offloading problem as a Markovian decision process and propose a solution employing deep reinforcement learning through a deep Q-network (DQN) approach.…”

Deep Reinforcement Learning for Resilient Power and Energy Systems: Progress, Prospects, and Future Avenues

“…In situations where immediate decisions are paramount, DRL can prove advantageous. For instance, in the context of cloud robotics, as demonstrated by Penmetcha et al [119], DRL-based dynamic computational offloading methods can yield rapid decisions, with a mean computation time of 71.28 milliseconds, while achieving a commendable final accuracy of 84%. This showcases the potential of DRL in scenarios where timely responses are essential.…”

Evaluating Distributed Computation Offloading Scalability for Multiple Robots

“…The latter are particularly relevant when robots or mobile devices must compete for limited offloading resources (whether the limit is due to computation resources or network connectivity limitations). Recently deep reinforcement learning approaches have been applied to learn the offloading decision-making process [23], [24].…”