An integrated perception pipeline for robot mission execution in unstructured environments

Narayanan, Priya; Yeh, Bryanna; Holmes, Emma; Martucci, Scott; Schmeckpeper, Karl; Mertz, Christoph; Osteen, Philip R.; Wigness, Maggie

doi:10.1117/12.2560699

Cited by 5 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Object Detection is performed using the maskrcnn-benchmark (Massa and Girshick, 2018) implementation of Faster-RCNN (Ren et al, 2015), trained with the RCTA dataset (Narayanan et al, 2020).…”

Section: Neural Network Approachmentioning

confidence: 99%

Human-Scale Mobile Manipulation Using RoMan

Kessens¹,

Kaplan²,

Rocks³

et al. 2022

View full text Add to dashboard Cite

We present the design, integration, and evaluation of a full-stack robotic system called RoMan, which can conduct autonomous field operations involving physical interaction with its environment. RoMan offers autonomous behaviors that can be triggered from succinct, high-level human input such as “open this box and retrieve the bag inside.” The robot’s behaviors are driven by a set of planners and controllers grounded in perceptual reconstructions of the environment. These behaviors are articulated by a behavior tree that translates high-level operator input into programs of increasing sensorimotor expressiveness, ultimately driving the lowest-level controllers. The software system is implemented in ROS as a set of independent processes connected by synchronous and asynchronous communication, and distributed across two on-board planning/control computers. The behavior stack drives a novel platform consisting of a pair of custom, 500 Nm/axis manipulators mounted on a rotatable torso aboard a tracked platform. The robot’s head is equipped with forward-looking depth cameras, and the arms carry wrist-mounted force-torque sensors and a mix of three- and four-finger grippers. We discuss design and implementation trade-offs affecting the entire hardware-software stack and high-level manipulation behaviors. We also demonstrate the applicability of the system for solving two manipulation tasks: 1) removing heavy debris from a roadway, where 64% of end-to-end autonomous runs required at most one human intervention; and 2) retrieving an item from a closed container, with a fully autonomous success rate of 56%. Finally, we indicate lessons learned and suggest outstanding research problems.

show abstract

Section: Neural Network Approachmentioning

confidence: 99%

Human-Scale Mobile Manipulation Using RoMan

Kessens¹,

Kaplan²,

Rocks³

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The keypoint-based pose estimation pipeline was used by (Bowman et al, 2017) as the first step in their semantic SLAM pipeline. Our pipeline has been used by (Vasilopoulos and Koditschek, 2018;Vasilopoulos et al, 2020b;Vasilopoulos et al, 2020a) to detect and localize objects for reactive planning for navigation, and was used by the Robotics Collaborative Techonology Alliance (RCTA) program to rapidly collect and annotate data for pose estimation for mobile manipulation (Narayanan et al, 2020;Kessens et al, 2020).…”

Section: Related Workmentioning

confidence: 99%

“…As part of this work, we have therefore created a dataset containing 101 sequences split across six of the objects used by the RCTA (Narayanan et al, 2020) in both indoor and outdoor environments: barrel, barrier, crate, gas can, Czech Hedgehog, and robot. The number of keypoints for each object class are shown in Table 1.…”

Section: Rcta Object Keypoints Datasetmentioning

confidence: 99%

Semantic keypoint-based pose estimation from single RGB frames

Schmeckpeper¹,

Osteen²,

Wang³

et al. 2022

Self Cite

View full text Add to dashboard Cite

This paper presents an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. Unlike prior investigators, we are agnostic to whether the object is textured or textureless, as the convnet learns the optimal representation from the available training-image data. Furthermore, the approach can be applied to instanceand class-based pose recovery. Additionally, we accompany our main pipeline with a technique for semi-automatic data generation from unlabeled videos. This procedure allows us to train the learnable components of our method with minimal manual intervention in the labeling process. Empirically, we show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios even against a cluttered background. We apply our approach both to several, existing, large-scale datasets - including PASCAL3D+, LineMOD-Occluded, YCB-Video, and TUD-Light - and, using our labeling pipeline, to a new dataset with novel object classes that we introduce here. Extensive empirical evaluations show that our approach is able to provide pose estimation results comparable to the state of the art.

show abstract

“…for object detection in RGB images, specifically the maskrcnn-benchmark implementation (Massa and Girshick, 2018). The object detector is trained to detect the RCTA specific object classes, using data from (Narayanan et al, 2020). Figure 3 visualizes example detections of some of these object classes.…”

Section: Semantic Perception and Simultaneous Localization And Mappingmentioning

confidence: 99%

An Intelligence Architecture for Grounded Language Communication with Field Robots

Howard¹,

Stump²,

Fink³

et al. 2022

Self Cite

View full text Add to dashboard Cite

For humans and robots to collaborate effectively as teammates in unstructured environments, robots must be able to construct semantically rich models of the environment, communicate efficiently with teammates, and perform sequences of tasks robustly with minimal human intervention, as direct human guidance may be infrequent and/or intermittent. Contemporary architectures for human-robot interaction often rely on engineered human-interface devices or structured languages that require extensive prior training and inherently limit the kinds of information that humans and robots can communicate. Natural language, particularly when situated with a visual representation of the robot’s environment, allows humans and robots to exchange information about abstract goals, specific actions, and/or properties of the environment quickly and effectively. In addition, it serves as a mechanism to resolve inconsistencies in the mental models of the environment across the human-robot team. This article details a novel intelligence architecture that exploits a centralized representation of the environment to perform complex tasks in unstructured environments. The centralized environment model is informed by a visual perception pipeline, declarative knowledge, deliberate interactive estimation, and a multimodal interface. The language pipeline also exploits proactive symbol grounding to resolve uncertainty in ambiguous statements through inverse semantics. A series of experiments on three different, unmanned ground vehicles demonstrates the utility of this architecture through its robust ability to perform language-guided spatial navigation, mobile manipulation, and bidirectional communication with human operators. Experimental results give examples of component-level behaviors and overall system performance that guide a discussion on observed performance and opportunities for future innovation.

show abstract

An integrated perception pipeline for robot mission execution in unstructured environments

Cited by 5 publications

References 0 publications

Human-Scale Mobile Manipulation Using RoMan

Human-Scale Mobile Manipulation Using RoMan

Semantic keypoint-based pose estimation from single RGB frames

An Intelligence Architecture for Grounded Language Communication with Field Robots

Contact Info

Product

Resources

About