Benchmarking Simulated Robotic Manipulation Through a Real World Dataset

Abstract

We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset. Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset. The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies.We apply our benchmark to two simulation environments, PyBullet and V-Rep, and publish the results. All materials required to benchmark an environment, including protocols and the dataset, can be found at the benchmarks' website 1 .

Traversing the Reality Gap via Simulator Tuning

“…Our dataset is a publicly available collection of tasks completed by a robotic manipulator and recorded by a motion capture system [2]. The data gives a ground truth of the real world with 6DOF pose of the manipulator and manipulated objects recorded.…”

“…Another useful property of the dataset is the contrast in object materials, with half the interaction tasks completed with plastic objects and the other half with exact wooden replicas. For a complete description of the dataset and its use in benchmarking reality, please see [2].…”

“…Our experimentation involves optimising two popular rigid body simulators used by the robotics community to faithfully replicate the results of a series of tasks conducted by a real robot in a motion capture system. The tasks are a range of kinematic movements and object interactions performed by a robotic manipulator, and the results of this 'ground truth' are available as a publiclyaccessible dataset [2]. Optimisation of simulator parameters is via differential evolution, an Evolutionary Algorithm that performs well on high-dimensional optimisation problems.…”

Guest Editorial: Introduction to the Special Issue on Benchmarking Protocols for Robotic Manipulation

“…The work in [13] focuses on dynamic human-to-robot handovers, and provides experimental procedures for transferring cups with different fullness levels and transparencies. The work in [14] provides a framework to assess the performance of robotic manipulation simulators via a real-world dataset.…”

Crossing The Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamics

“…Our work focuses on zero-shot sim-toreal transfer for dynamics, and we review in more detail closely related works below. Although existing works have compared the accuracy of different physics simulators, in sim-to-sim [11] or sim-to-real settings [9], to the best of our knowledge ours is the first to perform an extensive evaluation of sim-to-real transfer methods for dynamics.…”