Learning Navigation Costs from Demonstration in Partially Observable Environments

Wang, Tianyu; Dhiman, Vikas; Atanasov, Nikolay

doi:10.1109/icra40945.2020.9197199

Cited by 9 publications

(14 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reinforcement Learning (RL) can allow autonomous systems to learn policies for complex tasks, such as control [12] and obstacle avoidance [13], reducing the engineering effort to the design of a suitable reward function [14]. In combination with deep neural networks, deep RL provides a powerful tool capable of mapping high-dimensional sensory inputs to optimal actions, showing promising results in fields, such as autonomous driving [15] and robot navigation [16], [17].…”

Section: Related Workmentioning

confidence: 99%

Semantic-aware Active Perception for UAVs using Deep Reinforcement Learning

Bartolomei

Teixeira

Chli

2021

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

This work presents a semantic-aware pathplanning pipeline for Unmanned Aerial Vehicles (UAVs) using deep reinforcement learning for vision-based navigation in challenging environments. Driven by the maturity of works in semantic segmentation, the proposed path-planning architecture uses reinforcement learning to distinguish the parts of the scene that are perceptually more informative using semantic cues, in effect guiding more robust, repeatable, and accurate navigation of the UAV to the predefined goal destination. Assuming that the UAV performs vision-based state estimation, such as keyframebased visual odometry, and semantic segmentation onboard, the proposed deep policy network continuously evaluates the optimal relative perceptual informativeness of each semantic class in view. A perception-aware path planner uses these informativeness values to perform trajectory optimization in order to generate the next best action with respect to the current state and the perception quality of the surroundings, essentially guiding the UAV to avoid flying over perceptually degraded regions. Thanks to the use of semantic cues, the policy can be trained in a large number of non-photorealistic randomly-generated scenes, and results to an architecture that is generalizable to environments with the same semantic classes, independently of their visual appearance. Extensive evaluations on challenging, photorealistic simulations reveal a remarkable improvement in robustness and success rate with the proposed approach over the state of the art in active perception.Videohttps://youtu.be/RaO3whUBVnc

show abstract

Section: Related Workmentioning

confidence: 99%

Semantic-aware Active Perception for UAVs using Deep Reinforcement Learning

Bartolomei

Teixeira

Chli

2021

2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

show abstract

“…For example, α → ∞ means that the expert takes strictly optimal controls while α = 0 means random controls are selected. The Boltzmann expert model was previously introduced and studied in [30], [33], [34].…”

Section: B Demonstrator Modelmentioning

confidence: 99%

“…We discuss how to differentiate the loss function L c (θ) in ( 6) with respect to θ through the deterministic shortest path problem defined by the product WFA-MDP model. Wang et al [30] introduce a sub-gradient descent approach to differentiate the log likelihood of the expert demonstrations evaluated by the Bolzman policy in (5) through the optimal cost-to-go values in (7). The cost parameters can be updated by stochastic subgradient descent at each iteration k with learning rate γ (k) :…”

Section: Optimizing Cost Parametersmentioning

confidence: 99%

“…We propose a planning algorithm to search over the WFA-MDP product space for a policy that minimizes the cumulative transition cost estimates while satisfying task requirements specified by the learned WFA. Since the true cost is not directly observable, we differentiate the loss between the agent policy and the demonstrated controls through the planning algorithm to update the cost parameters using a subgradient method introduced in [30]. Evaluation in a Minigrid environment [10] (Fig.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

WFA-IRL: Inverse Reinforcement Learning of Autonomous Behaviors Encoded as Weighted Finite Automata

Wang¹,

Atanasov²

2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper presents a method for learning logical task specifications and cost functions from demonstrations. Linear temporal logic (LTL) formulas are widely used to express complex objectives and constraints for autonomous systems. Yet, such specifications may be challenging to construct by hand. Instead, we consider demonstrated task executions, whose temporal logic structure and transition costs need to be inferred by an autonomous agent. We employ a spectral learning approach to extract a weighted finite automaton (WFA), approximating the unknown logic structure of the task. Thereafter, we define a product between the WFA for high-level task guidance and a Labeled Markov decision process (L-MDP) for low-level control and optimize a cost function that matches the demonstrator's behavior. We demonstrate that our method is capable of generalizing the execution of the inferred task specification to new environment configurations.

show abstract

“…This paper is a revised and extended version of our previous conference publications (Wang et al 2020a,b). In our previous work (Wang et al 2020a), we proposed differentiable mapping and planning stages to learn the expert cost function. The cost function is parameterized as a neural network over binary occupancy probabilities, updated from local distance observations.…”

Section: Mapping and Planningmentioning

confidence: 99%

Inverse reinforcement learning for autonomous navigation via differentiable semantic mapping and planning

Wang¹,

Dhiman²,

Atanasov³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper focuses on inverse reinforcement learning for autonomous navigation using distance and semantic category observations. The objective is to infer a cost function that explains demonstrated behavior while relying only on the expert's observations and state-control trajectory. We develop a map encoder, that infers semantic category probabilities from the observation sequence, and a cost encoder, defined as a deep neural network over the semantic features. Since the expert cost is not directly observable, the model parameters can only be optimized by differentiating the error between demonstrated controls and a control policy computed from the cost estimate. We propose a new model of expert behavior that enables error minimization using a closed-form subgradient computed only over a subset of promising states via a motion planning algorithm. Our approach allows generalizing the learned behavior to new environments with new spatial configurations of the semantic categories. We analyze the different components of our model in a minigrid environment. We also demonstrate that our approach learns to follow traffic rules in the autonomous driving CARLA simulator by relying on semantic observations of buildings, sidewalks, and road lanes.

show abstract

Learning Navigation Costs from Demonstration in Partially Observable Environments

Cited by 9 publications

References 10 publications

Semantic-aware Active Perception for UAVs using Deep Reinforcement Learning

Semantic-aware Active Perception for UAVs using Deep Reinforcement Learning

WFA-IRL: Inverse Reinforcement Learning of Autonomous Behaviors Encoded as Weighted Finite Automata

Inverse reinforcement learning for autonomous navigation via differentiable semantic mapping and planning

Contact Info

Product

Resources

About