Table-top scene analysis using knowledge-supervised MCMC

Abstract

models Semantic modelling a b s t r a c tIn this paper, we propose a probabilistic approach to generate abstract scene graphs from uncertain 6D pose estimates. We focus on generating a semantic understanding of the perceived scenes that well explains the composition of the scene and the inter-object relations. The proposed system is realized by our knowledge-supervised MCMC sampling technique. We explicitly make use of task-specific context knowledge by encoding this knowledge as descriptive rules in Markov logic networks. We use a probabilistic sensor model to encode the fact that measurements are subject to significant uncertainty. We integrate the measurements with the abstract scene graph in a data driven MCMC process. Our system is fully probabilistic and links the high-level abstract scene description to uncertain low level measurements. Moreover, false estimates of the object poses and hidden objects of the perceived scenes can be systematically detected using the defined Markov logic knowledge base. The effectiveness of our approach is demonstrated and evaluated in real world experiments.

Goal-directed robot manipulation through axiomatic scene estimation

“…Avoiding exploration over all possible scenes, another approach to AxScEs inference is to search over scenes with algorithms amenable to general data structures, such as a hill climbing optimization or MCMC algorithm. Similar to Liu et al (2015), such an inference procedure samples over possible scenes g t where pose estimation on q t , on robot observations z t , is performed for each sampled scene. Our proposed axiomatic scene estimation by MCMC sampling (AxScEs MCMCs) takes this form.…”

“…AxMC performs scene inference of g t with the MCMC-based Metropolis–Hastings algorithm and pose inference of q t with a particle filter. AxMC works directly with depth images without the need for discriminative features, as used by Liu et al (2015) or Collet et al (2011). Further, AxMC provides distributions over both scene structure and object poses, which conceptually allows for update over time as we consider future work.…”

“…The AxMC method we propose in this article is very similar to the recently proposed knowledge-supervised (KS) MCMC method of Liu et al (2015). KSMCMC uses MCMC to sample over scene graph structures represented axiomatically to estimate objects as oriented bounding boxes.…”

Semantic Robot Programming for Goal-Directed Manipulation in Cluttered Scenes

“…Given observations of the scene, our work estimates a scene graph that represent the scene structure. Liu et al [23] also estimate a scene graph given observations, however, their approach approximates objects as oriented bounding boxes. Sui et al proposed a generative approach (AxMC) [35] for scene graph estimation and use Markov Chain Monte Carlo (MCMC) to search for the best scene graph hypothesis that explains the observations.…”

GRIP: Generative Robust Inference and Perception for Semantic Robot Manipulation in Adversarial Environments

“…The efficiency of deterministic inference without its fragility to uncertainty [11], [12]. Generative-discriminative algorithms may be especially advantageous when exposed to adversarial attack, building on foundational ideas in this space [13], [14], [15], [16], [17]. Furthermore, we expect our approach will be more generally applicable to guard against broad categories of attack with a clear pathway for explanability of the resulting perceptual estimates.…”