Hierarchical Semantic Labeling for Task-Relevant RGB-D Perception

Abstract

Abstract-Semantic labeling of RGB-D scenes is very important in enabling robots to perform mobile manipulation tasks, but different tasks may require entirely different sets of labels. For example, when navigating to an object, we may need only a single label denoting its class, but to manipulate it, we might need to identify individual parts. In this work, we present an algorithm that produces hierarchical labelings of a scene, following is-part-of and is-type-of relationships. Our model is based on a Conditional Random Field that relates pixel-wise and pair-wise observations to labels. We encode hierarchical labeling constraints into the model while keeping inference tractable. Our model thus predicts different specificities in labeling based on its confidence-if it is not sure whether an object is Pepsi or Sprite, it will predict soda rather than making an arbitrary choice. In extensive experiments, both offline on standard datasets as well as in online robotic experiments, we show that our model outperforms other stateof-the-art methods in labeling performance as well as in success rate for robotic tasks.

Modeling High-Dimensional Humans for Activity Anticipation using Gaussian Process Latent CRFs

“…Many variants augment CRFs with latent variables in order to model hidden states, such as latent CRFs [33] that have been applied to object recognition [36,14,42], scene understanding [35], gesture recognition [40] and grounding natural language to robotic tasks [29]. However, in these models, the predefined latent space is discrete and small to keep the learning and inference tractable.…”

Tell Me Dave: Context-Sensitive Grounding of Natural Language to Manipulation Instructions

“…In the area of computer vision, some works have considered relating phrases and attributes to images and videos [39,15,26,25,21,50]. These works focus primarily on labeling the image/video by modeling the rich perceptual data rather than modeling the relations in the language and the entities in the environment.…”

3D Object Detection and Scene Optimization for Tangible Augmented Reality

“…While most methods use 2D visual information only [2], there are numerous 3D shape based recognition techniques [3,4], as well as methods that use both visual and shape information [5,6]. Object detection methods are essential for scene understanding [7], which has a number of applications in different fields, such as robotics [1] or augmented reality [8].…”