CMAX++ : Leveraging Experience in Planning and Execution using Inaccurate Models

Abstract

Given access to accurate dynamical models, modern planning approaches are effective in computing feasible and optimal plans for repetitive robotic tasks. However, it is difficult to model the true dynamics of the real world before execution, especially for tasks requiring interactions with objects whose parameters are unknown. A recent planning approach, CMAX, tackles this problem by adapting the planner online during execution to bias the resulting plans away from inaccurately modeled regions. CMAX, while being provably guaranteed to reach the goal, requires strong assumptions on the accuracy of the model used for planning and fails to improve the quality of the solution over repetitions of the same task. In this paper we propose CMAX++, an approach that leverages real-world experience to improve the quality of resulting plans over successive repetitions of a robotic task. CMAX++ achieves this by integrating model-free learning using acquired experience with model-based planning using the potentially inaccurate model. We provide provable guarantees on the completeness and asymptotic convergence of CMAX++ to the optimal path cost as the number of repetitions increases. CMAX++ is also shown to outperform baselines in simulated robotic tasks including 3D mobile robot navigation where the track friction is incorrectly modeled, and a 7D pick-and-place task where the mass of the object is unknown leading to discrepancy between true and modeled dynamics.

Monte Carlo Tree Search in the Presence of Transition Uncertainty

“…Therefore, with the same amount of resources, we should be able to learn a more accurate uncertainty model than a transition model. (Vemula et al 2020;Vemula, Bagnell, and Likhachev 2021) also mention the same intuition, which motivated their approach. To illustrate this with an example, we designed a toy environment called the 2-Way GridWorld.…”

“…While previous approaches to using search with imperfect models exist (Vemula et al 2020;Vemula, Bagnell, and Likhachev 2021), to the best of our knowledge, there is no prior work that directly adapts MCTS to deal with model uncertainty. In our work, we define transition uncertainty as a measure of difference between the state transitions in the perfect model and in the model that is available to the agent.…”

Improved nonlinear model‐free adaptive iterative learning control in DoS attack environment

“…Because LQR control can be seen as model‐dependent RL method, RL can also be interpreted as data‐driven LQR control. [29] presents a theoretical study for the performance of both ILC and misspecified model on LQR problems with unknown transition dynamics, where ILC can be seen as LQR under the worst case modeling error, that is to say ILC is also a model‐independent LQR.…”