Exploiting domain knowledge for Object Discovery

Abstract

Abstract-In this paper, we consider the problem of Lifelong Robotic Object Discovery (LROD) as the long-term goal of discovering novel objects in the environment while the robot operates, for as long as the robot operates. As a first step towards LROD, we automatically process the raw video stream of an entire workday of a robotic agent to discover objects.We claim that the key to achieve this goal is to incorporate domain knowledge whenever available, in order to detect and adapt to changes in the environment. We propose a general graph-based formulation for LROD in which generic domain knowledge is encoded as constraints. Our formulation enables new sources of domain knowledge-metadata-to be added dynamically to the system, as they become available or as conditions change. By adding domain knowledge, we discover 2.7× more objects and decrease processing time 190 times. Our optimized implementation, HerbDisc, processes 6 h 20 min of RGBD video of real human environments in 18 min 30 s, and discovers 121 correct novel objects with their 3D models. I. INTRODUCTIONWe focus on the problem of Lifelong Robotic Object Discovery (LROD): discovering new objects unsupervised in an environment while the robot operates, for as long as the robot operates. Key challenges in LROD are scalability to massive datasets, robustness in dynamic human environments, and adaptability to different conditions over time. As a first step towards LROD, we address these challenges on an entire workday of data, collected as our robot explored a cluttered and populated office building. We show for the first time a system that processes, in under 19 minutes, hundreds of thousands of samples and over 6 h of continous exploration, to discover over a hundred new objects in cluttered human environments.Our key insight to making LROD feasible is to incorporate domain knowledge. More precisely, the data gathered by a service robot is not an unordered collection of anonymous images: range data is also available; we may know where and/or when the images are captured, as well as their ordering; we may know where interesting objects usually appear for a particular environment, such as in tables or cabinets; we may have additional sensing (e.g., robot localization, odometry) for some or all the images; or we may only be interested in objects of certain sizes or shapes relevant to the robot. In our work, we define the term metadata to encode any source of domain knowledge. Our definition includes any additional robotic sensing, assumptions, or prior information; in short, any non-visual data that can provide information about the object candidates.Prior work in robotics has widely used metadata to limit computational costs and improve robustness in perception. The metadata is mostly incorporated by imposing restrictions on the environment, data acquisition, or agent motion, which

Saliency-based object discovery on RGB-D data with a late-fusion approach

“…Although the dataset was not designed for object discovery it is to our knowledge the most suitable RGB-D dataset with annotated ground that is freely available. 4 On both datasets, we measure precision as the number of correct object candidates over the total number generated, and recall as the number of correct candidates over the total present in the ground truth. We consider candidates as correct if they satisfy the Pascal criterion, i.e., intersection over union is greater than 0.5.…”

“…For example, Herbst et al [11] discover objects by analysing the changes that have occurred in a scene at different points in time: elements that do not match to the 3D reconstruction of the scene are most likely objects. Collet et al [4] incorporate domain knowledge to support the discovery process. Other researches [16], [28] used interaction with the scene do detect objects as parts of the space that are moving together when pushed or carried around.…”

Simultaneous Recognition and Modeling for Learning 3-D Object Models From Everyday Scenes

“…It can be considered as the 3-D object discovery problem [21], which is of great interest, especially with the quick development of a RGB-D camera [22], [23]. Anguelov et al [24] proposed a hierarchical generative model for environments with nonstationary objects and employ map differencing technique to detect changes in the environment over time.…”

Data-driven exemplar model selection

“…For example, a natural way of training vision systems would be to present them with training samples from a large set of videos, e.g., as acquired from discovery/always-on/wearable systems like first-person-vision cameras, surveillance cameras, robotics systems [4,24], instead of providing a small set of individual frames. Such systems would have to deal with hundreds of thousands of samples, for which intelligent detector selection is imperative.…”