MotionBenchMaker: A Tool to Generate and Benchmark Motion Planning Datasets

Abstract

Recently, there has been a wealth of development in motion planning for robotic manipulation-new motion planners are continuously proposed, each with their own unique strengths and weaknesses. However, evaluating new planners is challenging and researchers often create their own ad-hoc problems for benchmarking, which is time-consuming, prone to bias, and does not directly compare against other state-of-the-art planners. We present MOTIONBENCHMAKER, an open-source tool to generate benchmarking datasets for realistic robot manipulation problems. MOTIONBENCHMAKER is designed to be an extensible, easyto-use tool that allows users to both generate datasets and benchmark them by comparing motion planning algorithms. Empirically, we show the benefit of using MOTIONBENCHMAKER as a tool to procedurally generate datasets which helps in the fair evaluation of planners. We also present a suite of 40 prefabricated datasets, with 5 different commonly used robots in 8 environments, to serve as a common ground to accelerate motion planning research.