APEX: Unsupervised, Object-Centric Scene Segmentation and Tracking for Robot Manipulation

Abstract

Recent advances in unsupervised learning for object detection, segmentation, and tracking hold significant promise for applications in robotics. A common approach is to frame these tasks as inference in probabilistic latent-variable models. In this paper, however, we show that the current state-of-the-art struggles with visually complex scenes such as typically encountered in robot manipulation tasks. We propose APEX, a new latent-variable model which is able to segment and track objects in more realistic scenes featuring objects that vary widely in size and texture, including the robot arm itself. This is achieved by a principled mask normalisation algorithm and a high-resolution scene encoder. To evaluate our approach, we present results on the real-world Sketchy dataset. This dataset, however, does not contain ground truth masks and object IDs for a quantitative evaluation. We thus introduce the Panda Pushing Dataset (P2D) which shows a Panda arm interacting with objects on a table in simulation and which includes groundtruth segmentation masks and object IDs for tracking. In both cases, APEX comprehensively outperforms the current state-ofthe-art in unsupervised object segmentation and tracking. We demonstrate the efficacy of our segmentations for robot skill execution on an object arrangement task, where we also achieve the best or comparable performance among all the baselines.

Efficient Skill Acquisition for Complex Manipulation Tasks in Obstructed Environments

“…Our work leverages OCGMs [12], [30], [31] to find a target object for MP, negating the need for object-specific goal estimators. OCGMs learn structured representations of objects within complex scenes and provide object-specific encodings useful for matching, however, most OCGMs have never been applied to real world environments.…”

“…template matching or object classifier, OCGMs hold the promise of versatile target object identification. This work leverages APEX [12] by training the model in an unsupervised manner on a wide distribution of simulated data to assist with generalisation to real-world environments.…”

“…Pre-training is required for APEX [12], our choice of unsupervised OCGM, to achieve versatile one-shot target object acquisition, but it only needs to be done once as it is trained on a diverse dataset to encourage generalise to a variety of real-world objects. APEX is formulated as a set of VAEs, and takes a video sequence I 1:T as input.…”

“…In this work, we take advantage of both MP and RL to achieve efficient skill acquisition for complex manipulation tasks in environments with arbitrary obstacles. We propose a system that leverages an object-centric generative model (OCGM) [12] for versatile, one-shot goal identification and re-identification as a key component in solving complex manipulation tasks from a few simple human demonstrations. Specifically, we identify a target object from a single demonstration using an OCGM, pre-trained on diverse synthetic scenes, leading to robust re-identification of that object in new scenes by matching to its object-centric representation.…”

Autoencoder-based background reconstruction and foreground segmentation with background noise estimation

“…Although the task of automatic background subtraction has been studied for more than 30 years [62], it is still considered as an open problem due to the various challenges appearing in real ap-plications : illumination changes, high level of occlusion of the background, background motions caused by moving trees or water, challenging weather conditions, presence of shadows... Dynamic background reconstruction models have been used for background subtraction, but are now also implemented as components of unsupervised object detection and tracking models [22,27,64].…”