PAC-Bayes Control: Learning Policies that Provably Generalize to Novel Environments

Abstract

Our goal is to synthesize controllers for robots that provably generalize well to novel environments given a dataset of example environments. The key technical idea behind our approach is to leverage tools from generalization theory in machine learning by exploiting a precise analogy (which we present in the form of a reduction) between robustness of controllers to novel environments and generalization of hypotheses in supervised learning. In particular, we utilize the Probably Approximately Correct (PAC)-Bayes framework, which allows us to obtain upper bounds (that hold with high probability) on the expected cost of (stochastic) controllers across novel environments. We propose control synthesis algorithms that explicitly seek to minimize this upper bound. The corresponding optimization problem can be solved using convex optimization (Relative Entropy Programming in particular) in the setting where we are optimizing over a finite control policy space. In the more general setting of continuously parameterized controllers, we minimize this upper bound using stochastic gradient descent. We present examples of our approach in the context of obstacle avoidance control with depth measurements.Our simulated examples demonstrate the potential of our approach to provide strong generalization guarantees on controllers for robotic systems with continuous state and action spaces, complicated (e.g., nonlinear) dynamics, and rich sensory inputs (e.g., depth measurements).

Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning

“…On the theoretical front, an important direction for future work is to provide rigorous guarantees on generalization to novel domains. One potential avenue is to combine the algorithmic techniques presented here with recent results on PAC-Bayes generalization theory applied to control and RL settings [13,36]. On the algorithmic front, an interesting direction is to use domain randomization techniques to automatically generate new training domains that can be used to improve invariant policy learning (e.g., automatically generating domains with different colored keys in the colored-keys example).…”

“…Distributional robustness. The PAC-Bayes Control approach [13,14] provides a way to make provable generalization guarantees under distributional shifts. This approach is particularly useful in safety-critical applications where it is important to quantify the impact of switching between a training domain and a test domain.…”

Integrative Use of Computer Vision and Unmanned Aircraft Technologies in Public Inspection

“…Majumdar et al explore one potential theory for an automated guidance system, which leverages another form of machine learning. Majumdar et al utilizes a similar framework for both the navigation of a UAS, and the task of grasping an object [10]. One issue with the grasping framework presented in the paper can be found in the mass measurement used for crucial grasping calculations [10].…”

“…Majumdar et al utilizes a similar framework for both the navigation of a UAS, and the task of grasping an object [10]. One issue with the grasping framework presented in the paper can be found in the mass measurement used for crucial grasping calculations [10]. The mass used in their calculations is a randomly generated number in the range [0.05, 0.15] kg [10], which could cause inaccurate results in a real setting.…”

“…One issue with the grasping framework presented in the paper can be found in the mass measurement used for crucial grasping calculations [10]. The mass used in their calculations is a randomly generated number in the range [0.05, 0.15] kg [10], which could cause inaccurate results in a real setting. This problem could be solved by using the suggested estimated mass calculation presented to achieve a more tailored result.…”