Scene Understanding through Autonomous Interactive Perception

Abstract

Abstract. We propose a framework for detecting, extracting and modeling objects in natural scenes from multi-modal data. Our framework is iterative, exploiting different hypotheses in a complementary manner. We employ the framework in realistic scenarios, based on visual appearance and depth information. Using a robotic manipulator that interacts with the scene, object hypotheses generated using appearance information are confirmed through pushing. The framework is iterative, each generated hypothesis is feeding into the subsequent one, continuously refining the predictions about the scene. We show results that demonstrate the synergic effect of applying multiple hypotheses for real-world scene understanding. The method is efficient and performs in real-time.

Probabilistic Segmentation and Targeted Exploration of Objects in Cluttered Environments

“…If 3D-data is available, 3D features such as surface normals or curvature might additionally be exploited [10,11]. However, visual or spatial boundaries need not always correspond to object boundaries [12,13], so not all ambiguities can be resolved [12,[14][15][16]]. An alternative is to look at video streams [17]; however, in real-robot setups there may be too much (self-)occlusion for this strategy to be viable.…”

“…In a subsequent stage, movement and object membership can be estimated for each of these parts. Alternative approaches use algorithms such as iterative closest point (ICP) to determine whether the tracked point cloud is a single rigid object [12,31], or estimate the movement of trackable visual features [13,16,25,[31][32][33].…”

“…Many of the discussed approaches do not handle occlusion and co-movement as they deal with only one object of interest, e.g., [6,7,15]. Noise is often ignored [14,15] or handled by requiring objects to move as rigid bodies during one or multiple actions [12,16,19,33]. In these approaches, it is not clear how to deal with uncertainty, e.g., from occlusions or with pushes resulting in multiple adjacent objects moving as according to the same homogeneous transform.…”

“…Bergström et al [16] combined rigid motion clues with color-and disparity clues, whereas Schiebener et al [25] validated hypotheses based on proximity and shared parametric surfaces using co-movement. Hausman et al [33] used visual features to generate hypotheses, and to reconstruct a dense model from clustered feature points.…”

“…The methods discussed in the previous paragraph used visual features to create hypotheses that were tested by interaction [16,25,33] or to modify binary potentials between parts [13]. Instead, a probabilistic approach offers a principled way to integrate noisy clues from multiple sources.…”