Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges

Abstract

The Internet of Things (IoT) extends the Internet connectivity into billions of IoT devices around the world, which collect and share information to reflect the status of physical world. The Autonomous Control System (ACS), on the other hand, performs control functions on the physical systems without external intervention over an extended period of time. The integration of IoT and ACS results in a new concept -autonomous IoT (AIoT). The sensors collect information on the system status, based on which intelligent agents in IoT devices as well as Edge/Fog/Cloud servers make control decisions for the actuators to react. In order to achieve autonomy, a promising method is for the intelligent agents to leverage the techniques in the field of artificial intelligence, especially reinforcement learning (RL) and deep reinforcement learning (DRL) for decision making. In this paper, we first provide comprehensive survey of the state-of-art research, and then propose a general model for the applications of RL/DRL in AIoT. Finally, the challenges and open issues for future research are identified.

Deep Reinforcement Learning for Smart Home Energy Management

“…Since DRL problems are mainly based on Markov Decision Process (MDP) framework or its variants (e.g., Partially observable MDP [30], Markov games [17]), we first introduce the background of MDP. Typically, an MDP is defined by a fivetuple (S, A, P, R, γ), where S and A denote the sets of state and action, respectively.…”

“…In the above research efforts, the proposed DQN-based methods can not deal with DRL problems with continuous actions, e.g., the generation output of Diesel Generators (DG) [30]. To support continuous actions, DDPG-based methods could be adopted.…”

“…To support continuous actions, DDPG-based methods could be adopted. For example, Lei et al proposed a FH-DDPG based energy management algorithm for an isolated microgrid to minimize the sum of power generation cost and the power unbalance penalty [30]. Since model-free based DRL algorithms in existing works have low data efficiency, Shuai et al proposed a model-based DRL algorithm (i.e., MuZero) for the online scheduling of a residential microgrid under uncertainties [45] based on Monte-Carlo tree search (MCTS) strategy with a learned network model.…”

(Deep) Reinforcement learning for electric power system control and related problems: A short review and perspectives

“…Reference [116] discussed applications and challenges of DRL in this context and revealed opportunities to use DRL in all three layers of Internet of Things: perception layer (control of the physical system or its components), network layer (control of communications resources) and application layer (control of computation resources). Future considerations should take into account the use of blockchain technology in this context [117].…”

Age-Aware Status Update Control for Energy Harvesting IoT Sensors via Reinforcement Learning

“…Related works: RL is an online machine learning method which learns an optimal policy through the interactions between the agent (the edge node in our case) and the environment. A comprehensive survey of RL based methods for autonomous IoT networks is presented in [7]. In [8], [9], the authors used RL to find an optimal caching policy for non-transient data (e.g., multimedia files).…”