Safe Q-Learning Method Based on Constrained Markov Decision Processes

Abstract

The application of reinforcement learning in industrial fields makes the safety problem of the agent a research hotspot. Traditional methods mainly alter the objective function and the exploration process of the agent to address the safety problem. Those methods, however, can hardly prevent the agent from falling into dangerous states because most of the methods ignore the damage caused by unsafe states. As a result, most solutions are not satisfactory. In order to solve the aforementioned problem, we come forward with a safe Q-learning method that is based on constrained Markov decision processes, adding safety constraints as prerequisites to the model, which improves standard Q-learning algorithm so that the proposed algorithm seeks for the optimal solution ensuring that the safety premise is satisfied. During the process of finding the solution in form of the optimal state-action value, the feasible space of the agent is limited to the safe space that guarantees the safety via the feasible space being filtered by constraints added to the action space. Because the traditional solution methods are not applicable to the safe Q-learning model as they tend to obtain local optimal solution, we take advantage of the Lagrange multiplier method to solve the optimal action that can be performed in the current state based on the premise of linearizing constraint functions, which not only improves the efficiency and accuracy of the algorithm, but also guarantees to obtain the global optimal solution. The experiments verify the effectiveness of the algorithm. INDEX TERMS Constrained Markov decision processes, safe reinforcement learning, Q-learning, constraint, Lagrange multiplier.

Deep reinforcement learning in production systems: a systematic literature review

“…This primarily involves the consideration of safety-relevant parameters to avoid critical actions and threats. In this context, a constraint-driven approach in non-deep RL was proposed by Ge et al (2019), in which permitted actions were limited through preliminary filtering, or by Xiong and Diao (2021) who proposed a safety-based evaluation of policy robustness. Further studies should approximate the simulations and frameworks to real-world conditions even more, which includes consideration of hard real-time requirements, significant parameters, uncertainties, and indeterminacies.…”

Reinforcement Learning Based EV Charging Management Systems–A Review

“…There is no guarantee that an EV has a fully-charged battery at its time of departure set in a specific charging schedule because of the random behavior of the EVs in arrival and departure times. The authors in [100] address this issue by considering the EV problem as a constrained Markov Decision Process (CMDP) and solves the problem with constrained policy optimization (CPO) [122]. They propose a real-time strategy for charging and discharging the EVs that tackles the random behavior of EVs in arrival and departure, prices of electricity, and battery remnant energy.…”

MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning

“…In most cases, a stochastic (e.g., random) policy is selected as the behaviour policy to ensure enough exploration of new states. One of the most practiced off-policy methods is known as Q-learning [41], [42], [44], [46], which updates the value function using the Bellman optimality equation as follows…”