Reactive Semantic Planning in Unexplored Semantic Environments Using Deep Perceptual Feedback

Abstract

This paper presents a reactive planning system that enriches the topological representation of an environment with a tightly integrated semantic representation, achieved by incorporating and exploiting advances in deep perceptual learning and probabilistic semantic reasoning. Our architecture combines object detection with semantic SLAM, affording robust, reactive logical as well as geometric planning in unexplored environments. Moreover, by incorporating a human mesh estimation algorithm, our system is capable of reacting and responding in real time to semantically labeled human motions and gestures. New formal results allow tracking of suitably non-adversarial moving targets, while maintaining the same collision avoidance guarantees. We suggest the empirical utility of the proposed control architecture with a numerical study including comparisons with a state-of-the-art dynamic replanning algorithm, and physical implementation on both a wheeled and legged platform in different settings with both geometric and semantic goals.

Reactive navigation in partially familiar planar environments using semantic perceptual feedback

“…Similarly, sampling-based methods, motivated by the typically high-dimensional configuration spaces arising from combined task and motion planning (Garrett et al, 2018), can achieve asymptotic optimality (Vega-Brown and Roy, 2018), but no guarantee of convergence (or task completion) under partial prior knowledge or limited sampling, and their probabilistic completeness guarantees can be slow to be realized in practice when confronting settings with narrow passages (Noreen et al, 2016), even in 2D environments. More importantly, our recent parallel work (Vasilopoulos et al, 2020), that uses the reactive planning principles presented in this article, shows that existing state-of-the-art path replanning algorithms for unknown 2D environments (Otte and Frazzoli, 2015) can cycle repeatedly in the presence of both unforeseen obstacles and narrow passages as they search for alternative openings, before eventually (and after protracted cycling) reporting failure (incorrectly) and halting.…”

“…This is a first step toward integrating the reactive planning architecture developed in this work in the multi-layer architecture presented in , for accomplishing increasingly complicated mobile manipulation tasks with underactuated legged robots in environments that are semantically partially known and geometrically unknown. Parallel work, relying on the formal guarantees presented in this article, has already demonstrated how the same vector field planning principles and our semantic inference capabilities can be exploited in order to perform more complex missions with predefined logic that involve human following and pose tracking (Vasilopoulos et al, 2020), tasks such as Bottom: illustrations of the recorded semantic map and the robot's trajectory in RViz (Quigley et al, 2009). The robot detects and avoids the two chairs in front of it, though they are only temporarily included in the semantic map (in the absence of more frame measurements).…”

An Efficient Locally Reactive Controller for Safe Navigation in Visual Teach and Repeat Missions

“…In this work, we use a twist interface to control the legged robot's body velocity. It has been successfully demonstrated on several platforms such as ANYmal C [9], Mini Cheetah [10], and Vision60 [11]. Our controller is based on the Riemannian Motion Policies (RMP) framework [4], a recent approach to robot motion generation.…”

Hybrid dynamical type theories for navigation

“…We first provide a conjectural translation from a fragment of lineartime temporal logic (LTL) into the type theory with the goal of interfacing with existing state-of-the-art methods for controller synthesis from formal task specifications [22,34]. Then as a case study, we type various components of a navigational controller designed by Arslan-Koditschek [2] and implemented by Vasilopolous [37].…”