OLT: A Toolkit for Object Labeling applied to robotic RGB-D datasets

Abstract

Abstract-In this work we present the Object Labeling Toolkit (OLT), a set of software components publicly available for helping in the management and labeling of sequential RGB-D observations collected by a mobile robot. Such a robot can be equipped with an arbitrary number of RGB-D devices, possibly integrating other sensors (e.g. odometry, 2D laser scanners, etc.). OLT first merges the robot observations to generate a 3D reconstruction of the scene from which object segmentation and labeling is conveniently accomplished. The annotated labels are automatically propagated by the toolkit to each RGB-D observation in the collected sequence, providing a dense labeling of both intensity and depth images. The resulting objects' labels can be exploited for many robotic oriented applications, including high-level decision making, semantic mapping, or contextual object recognition. Software components within OLT are highly customizable and expandable, facilitating the integration of already-developed algorithms. To illustrate the toolkit suitability, we describe its application to robotic RGB-D sequences taken in a home environment.

Scene object recognition for mobile robots through Semantic Knowledge and Probabilistic Graphical Models

“…Therefore, a significant, growing body of current research aiming to overcome this issue is considering contextual information of the scene objects in addition to their usually employed individual features, and a number of applications dealing with this source of information have came out, e.g. Wong et al (2015) or Ruiz-Sarmiento, J.R. et al (2015b). Some works have attempted to exploit this information by providing ad-hoc or preliminary solutions, like in Mekhalfi et al (2015), where the co-occurrence of objects appearing in distinct types of rooms are implicitly modelled.…”

Robot@Home, a robotic dataset for semantic mapping of home environments

“…This tool is fed with both the recorded sequence and the labeled, reconstructed map (obtained as described in the previous section) in order to automatically propagate the ground truth information to the RGB-D observations. The outcome of this process is a per-pixel labeling of the intensity and depth images within each observation, as well as a per-point labeling of its point cloud data (please refer to Ruiz-Sarmiento et al (2015c) for further information). The last row of Figure 8 depicts depth images colored according to the propagated ground truth labels.…”

“…Processed data have been produced employing two software tools, namely the aforementioned Mobile Robot Programming Toolkit (MRPT), and the Object Labeling Toolkit (OLT) (Ruiz-Sarmiento et al, 2015c). OLT comprises a set of public tools 6 aimed at helping in the management and labeling of sequential RGB-D observations.…”

“…This is a valuable feature, missing in most of the state-of-the-art datasets, which can be exploited by, for instance, semantic mapping systems that leverage relationships such as pillows are usually on beds or ovens are not in bathrooms . Robot@Home is publicly available and is accompanied with the software application employed for its processing, named the Object Labeling Toolkit (OLT) (Ruiz-Sarmiento et al, 2015c).…”

Qualitative spatial logic descriptors from 3D indoor scenes to generate explanations in natural language

“…Moreover, 3D robust recognition of everyday objects have been achieved by applying different machine learning techniques: e.g., depth kernel descriptors (Bo et al, 2011b), hierarchical kernel descriptors (Bo et al, 2011a), sparse distance learning (Lai et al, 2011a), scalable and hierarchical recognition (Lai et al, 2011b) or multi-scene analysis (Herbst et al, 2011b). Among other things, RGB-D cameras have been used to recognize human poses (Shotton et al, 2011), to build and maintain semantic maps of scenes using probabilistic graphical models for recognizing objects and rooms (Ruiz-Sarmiento, 2016;Ruiz-Sarmiento et al, 2015), etc.…”