Causal Transfer for Imitation Learning and Decision Making under Sensor-Shift

Abstract

Learning from demonstrations (LfD) is an efficient paradigm to train AI agents. But major issues arise when there are differences between (a) the demonstrator's own sensory input, (b) our sensors that observe the demonstrator and (c) the sensory input of the agent we train.In this paper, we propose a causal model-based framework for transfer learning under such “sensor-shifts”, for two common LfD tasks: (1) inferring the effect of the demonstrator's actions and (2) imitation learning. First we rigorously analyze, on the population-level, to what extent the relevant underlying mechanisms (the action effects and the demonstrator policy) can be identified and transferred from the available observations together with prior knowledge of sensor characteristics. And we device an algorithm to infer these mechanisms. Then we introduce several proxy methods which are easier to calculate, estimate from finite data and interpret than the exact solutions, alongside theoretical bounds on their closeness to the exact ones. We validate our two main methods on simulated and semi-real world data.

Fast Counterfactual Inference for History-Based Reinforcement Learning

“…Instead, our approach counts the causal effects on future transition probabilities, as shown in our ATE losses that are built upon the dynamics model-invariant metrics (Tomar et al 2021). The methods of CI over time (Bica et al 2020;Zhang et al 2022a) have been developed, and recently a few studies prove that causal inference in partially observable environments is theoretically feasible in imitation learning domains (Kumor, Zhang, and Bareinboim 2021;Zhang, Kumor, and Bareinboim 2020;Etesami and Geiger 2020;de Haan, Jayaraman, and Levine 2019). However, these studies rely on sufficient expert data, which is not available in RL domains.…”

Learning Human Driving Behaviors with Sequential Causal Imitation Learning

“…Causal Imitation Learning (CIL) was recently proposed by (Zhang, Kumor, and Bareinboim 2020), which focuses on learning a policy within the limit of nonsequential one-stage settings. The authors of (Etesami and Geiger 2020) study the causal transfer problem by assuming that the relationships among variables are linear. The paper (de Haan, Jayaraman, and Levine 2019) ignores unobserved confounders and assumes the reward and the expert can be easily accessible.…”