Bi-Directional Value Learning for Risk-Aware Planning Under Uncertainty

Abstract

Decision-making under uncertainty is a crucial ability for autonomous systems. In its most general form, this problem can be formulated as a Partially Observable Markov Decision Process (POMDP). The solution policy of a POMDP can be implicitly encoded as a value function. In partially observable settings, the value function is typically learned via forward simulation of the system evolution. Focusing on accurate and long-range risk assessment, we propose a novel method, where the value function is learned in different phases via a bi-directional search in belief space. A backward value learning process provides a long-range and risk-aware base policy. A forward value learning process ensures local optimality and updates the policy via forward simulations. We consider a class of scalable and continuous-space rover navigation problems (RNP) to assess the safety, scalability, and optimality of the proposed algorithm. The results demonstrate the capabilities of the proposed algorithm in evaluating long-range risk/safety of the planner while addressing continuous problems with long planning horizons.Proof. We prove this by backward induction.Consider the terminal beliefs first. Trivially, from Eq. (4) and Eq. (8), ρ(b g ; π) = lim J F →∞ J(bg;π) J F = 0 for ∀b g ∈ B goal , and ρ(b f ; π) = lim J F →∞

Towards Energy-Aware Feedback Planning for Long-Range Autonomous Underwater Vehicles

“…, y t ). The belief state b t can be recursively updated with the following transition function τ (Kim et al, 2019)…”

“…Various types of rewards modification in POMDPs have been investigated in previous research efforts ( Lee et al, 2018 ; Kim et al, 2019 ). Typically, the reward function in POMDPs is designed to solve the stochastic shortest path problem, where the goal is to compute a feedback plan that reaches a target state from a known initial state by maximizing the expected total reward.…”

“…A belief state of the vehicle is defined as a posterior distribution over all possible states given the past actions and sensor observations . The belief state can be recursively updated with the following transition function τ ( Kim et al, 2019 ) in which the next belief state depends only on the current belief state, action, and observation.…”

Perception‐aware autonomous mast motion planning for planetary exploration rovers

“…For replanning purposes, forward search algorithms can be used in an RHC scheme. Recently, methods using RHC have been extended for the belief space as well as dynamic environments (Agha‐mohammadi, Agarwal, Kim, Chakravorty, & Amato, ; Chakravorty & Erwin, ; Erez & Smart, ; He, Brunskill, & Roy, ; Kim, Thakker, & Agha‐mohammadi, ; Platt, Tedrake, Kaelbling, & Lozano‐Perez, ; Toit & Burdick, ). In an RHC scheme, optimization is performed only within a limited horizon; thus, the system performs optimization within the specified horizon, then the system takes the next immediate action and moves the optimization horizon one step forward before repeating the process.…”

Where to Map? Iterative Rover-Copter Path Planning for Mars Exploration

“…For example, the works in [8], [9] improved the performance of VO localization by actively choosing timing and camera direction to obtain an optimal image sequence using the predictive perception technique. This problem is typically approached by Partially Observable Markov Decision Process (POMDP), or belief-space planning [10], [11], [12], [13], [14], where the planner chooses optimal actions under motion and sensing uncertainty.…”