Active Policy Learning for Robot Planning and Exploration under Uncertainty

Abstract

Abstract-This paper proposes a simulation-based active policy learning algorithm for finite-horizon, partially-observed sequential decision processes. The algorithm is tested in the domain of robot navigation and exploration under uncertainty, where the expected cost is a function of the belief state (filtering distribution). This filtering distribution is in turn nonlinear and subject to discontinuities, which arise because constraints in the robot motion and control models. As a result, the expected cost is non-differentiable and very expensive to simulate. The new algorithm overcomes the first difficulty and reduces the number of simulations as follows. First, it assumes that we have carried out previous evaluations of the expected cost for different corresponding policy parameters. Second, it fits a Gaussian process (GP) regression model to these values, so as to approximate the expected cost as a function of the policy parameters. Third, it uses the GP predicted mean and variance to construct a statistical measure that determines which policy parameters should be used in the next simulation. The process is iterated using the new parameters and the newly gathered expected cost observation. Since the objective is to find the policy parameters that minimize the expected cost, this active learning approach effectively trades-off between exploration (where the GP variance is large) and exploitation (where the GP mean is low). In our experiments, a robot uses the proposed method to plan an optimal path for accomplishing a set of tasks, while maximizing the information about its pose and map estimates. These estimates are obtained with a standard filter for SLAM. Upon gathering new observations, the robot updates the state estimates and is able to replan a new path in the spirit of openloop feedback control.

Mapping, Planning and Exploration with Pose SLAM

“…This idea has been pursued in some works [67,[94][95][96]106]. Huang et al [67] introduced a discussion about the problem of multi-step look-ahead exploration in the context of SLAM, arguing that multi-step active SLAM is possible when the current estimation error is small, the probability of observing new feature is low, and the computation capability is high.…”

“…In the work presented in [105,106], Martinez Cantin et al proposed a reinforcement learning approach to solve the problem of exploration for SLAM, their technique is based on the work presented in [114]. They employ a direct policy search approach [122], where the value funtion is approximated using Gaussian Processes (GP).…”

Gaussian Process models for ubiquitous user comfort preference sampling; global priors, active sampling and outlier rejection

“…Active learning differs however in that the aim is only to poll the user when the information returned is useful (above a threshold or according to some constrained budget). To the best of our knowledge is it novel in the area of thermal comfort modelling but has a long history and has been applied in many fields such as robot control [24], fault detection [25], as a general optimisation approach [26] amongst others [27][28][29]. 4 …”

“…Note that it is through ζ x that C1 and C2 may be considered in the model. The maximum a-posteriori estimates for the process parameters are [24]:…”

Action Selection for Single-Camera SLAM

“…In our case, the expected number of landmarks to see and a very rough uniform disposition of them in the environment are our initial conditions. Several authors make such assumption either with a priori grid-based discretization of the environment [14], [20] or by adding uniformly distributed unvisited landmarks as vague priors [21], [22].…”