Real-time Detection of 2D Tool Landmarks with Synthetic Training Data

Abstract

In this paper a deep learning architecture is presented that can, in real time, detect the 2D locations of certain landmarks of physical tools, such as a hammer or screwdriver. To avoid the labor of manual labeling, the network is trained on synthetically generated data. Training computer vision models on computer generated images, while still achieving good accuracy on real images, is a challenge due to the difference in domain. The proposed method uses an advanced rendering method in combination with transfer learning and an intermediate supervision architecture to address this problem. It is shown that the model presented in this paper, named Intermediate Heatmap Model (IHM), generalizes to real images when trained on synthetic data. To avoid the need for an exact textured 3D model of the tool in question, it is shown that the model will generalize to an unseen tool when trained on a set of different 3D models of the same type of tool. IHM is compared to two existing approaches to keypoint detection and it is shown that it outperforms those at detecting tool landmarks, trained on synthetic data.

Untitled

“…This shows that the CAD2Render toolkit is able to create synthetic images that are suitable for this problem space and that images produced by the faster hybrid rendering mode can produce good models as well. Additionally, research has shown that models trained on images generated by CAD2Render perform better on this task than models trained on images generated by simple 2D image augmentations [31]. This shows the benefit of using 3D information to generate training data.…”

“…For some applications a large number of images might be needed. Complementary research has experimented with the amount of CAD2Render images needed to train object detection models and has shown that a large amount of images are beneficial when no domain knowledge is used [20]. hybrid renderer regularly fails to show all reflections, shadows and highlights that would be present in images generated with the path tracer.…”

“…The most common parameters are the camera, object, material and light parameters. Here the default parameters can be used but adding domain knowledge in the data generation can drastically reduce the amount of data that is required to train the AI-models [20].…”

“…A study on the impact of these variations on downstream performance is beyond the scope of this paper. For an in depth analysis of the impact of light and pose variations on object detection performance, for datasets generated by the CAD2Render tool, we refer to complementary research [20].…”

Sim-to-Real Dataset of Industrial Metal Objects

“…The high fidelity of our synthetic images, their close correspondence with their realworld counterparts, and the controlled variations in the data generation process will be valuable for sim-to-real research. Recently, our dataset was used to train object detection models with synthetic data [33]. The authors trained a model on various sub-sets of our synthetic data to determine which variations improved model performance.…”