Safe Value Functions

Abstract

The relationship between safety and optimality in control is not well understood, and they are often seen as important yet conflicting objectives. There is a pressing need to formalize this relationship, especially given the growing prominence of learning-based methods. Indeed, it is common practice in reinforcement learning to simply modify reward functions by penalizing failures, with the penalty treated as a mere heuristic. We rigorously examine this relationship, and formalize the requirements for safe value functions: value functions that are both optimal for a given task, and enforce safety. We reveal the structure of this relationship through a proof of strong duality, showing that there always exists a finite penalty that induces a safe value function. This penalty is not unique, but upperunbounded: larger penalties do not harm optimality. Although it is often not possible to compute the minimum required penalty, we reveal clear structure of how the penalty, rewards, discount factor, and dynamics interact. This insight suggests practical, theory-guided heuristics to design reward functions for control problems where safety is important.

Learning Robust Output Control Barrier Functions From Safe Expert Demonstrations

“…Learning with CBFs: Approaches that use CBFs during learning typically assume that a valid CBF is already given, while we focus on constructing CBFs so that our approach can be viewed as complementary. In [19], it is shown how safe and optimal reward functions can be obtained, and how these are related to CBFs. The authors in [20] use CBFs to learn a provably correct neural network safety guard for kinematic bicycle models.…”

Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations

“…These works typically assume that a CBF is already given, while we in this paper focus on constructing CBFs so that our approach should be viewed as complementary. In [17], it is shown how safe and optimal reward functions can be obtained and how these are related to CBFs. The authors in [18] learn a provably correct neural network safety guard for kinematic bicycle models using CBFs as safety filters.…”

A Risk-Averse Preview-based $Q$-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles

“…Note that the RAU module encodes both safety and performance specifications in the reward function. It is shown in [64] that a big enough penalty function can assure that the optimal solution for the original task can be learned safely.…”