RB2: Robotic Manipulation Benchmarking with a Twist

Abstract

Benchmarks offer a scientific way to compare algorithms using objective performance metrics. Good benchmarks have two features: (a) they should be widely useful for many research groups; (b) and they should produce reproducible findings. In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility. If the benchmark is kept restrictive (fixed hardware, objects), the numbers are reproducible but the setup becomes less general. On the other hand, a benchmark could be a loose set of protocols (e.g. object set [9]) but the underlying variation in setups make the results non-reproducible. In this paper, we re-imagine benchmarking for robotic manipulation as state-of-the-art algorithmic implementations, alongside the usual set of tasks and experimental protocols. The added baseline implementations will provide a way to easily recreate SOTA numbers in a new local robotic setup, thus providing credible relative rankings between existing approaches and new work. However, these "local rankings" could vary between different setups. To resolve this issue, we build a mechanism for pooling experimental data between labs, and thus we establish a single global ranking for existing (and proposed) SOTA algorithms. Our benchmark, called Ranking-Based Robotics Benchmark (RB2), is evaluated on tasks that are inspired from clinically validated Southampton Hand Assessment Procedures [27]. Our benchmark was run across two different labs and reveals several surprising findings. For example, extremely simple baselines like open-loop behavior cloning, outperform more complicated models (e.g. closed loop, RNN, Offline-RL, etc.) that are preferred by the field. We hope our fellow researchers will use RB2 to improve their research's quality and rigor.

Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

“…Meta-World [49] reward functions are 50 lines! ), hampered by unstable learning dynamics [21], and poorly generalize in realistic robotic manipulation settings [12]. In contrast, our method learns a proxy for value functions (i.e.…”

Self-Driving Laboratories for Chemistry and Materials Science

“…81 Closer to tasks related to laboratory automation, RB2 proposed a robotics simulation benchmark with pouring, scooping, and insertion tasks. 82 In another work, a differentiable environment FluidLab 83 was proposed for simulating complex fluid manipulation tasks. An example of using digital twin for the SDL is provided in Vescovi et al, where simulated environments have been used to visualize and compare tools, verifying the laboratory operations.…”

“…In the realm of psychological safety, considerations such as robot motion, speed, adaptability, and appearance play pivotal roles in reducing stress and fostering a sense of safety in human-robot interactions. More information about robotics safety standards can be found in ISO 10218 82 , ISO/TS 15066 83 , and survey papers by Lasota et al 80 and Zacharaki et al 81 Additional safety issues may arise from the manipulation and perception capabilities of robotic systems, as these remain open problems in the community. When deploying such robotic systems in chemistry labs, if manipulation policies and the robot's decision-making abilities are not robust enough, it may lead to failures, increasing the risk of accidents.…”