Real-Time Joint-Stem Prediction for Agricultural Robots in Grasslands Using Multi-Task Learning

Abstract

Autonomous weeding robots need to accurately detect the joint stem of grassland weeds in order to control those weeds in an effective and energy-efficient manner. In this work, keypoints on joint stems and bounding boxes around weeds in grasslands are detected jointly using multi-task learning. We compare a two-stage, heatmap-based architecture to a single-stage, regression-based architecture—both based on the popular YOLOv5 object detector. Our results show that introducing joint-stem detection as a second task boosts the individual weed detection performance in both architectures. Furthermore, the single-stage architecture clearly outperforms its competitors with an OKS of 56.3 in joint-stem detection while also achieving real-time performance of 12.2 FPS on Nvidia Jetson NX, suitable for agricultural robots. Finally, we make the newly created joint-stem ground-truth annotations publicly available for the relevant research community.

Few-leaf Learning: Weed Segmentation in Grasslands

“…When plants have a fairly small size, joint stem detection enables automatic weeding, as for example in the work of Lottes et al [18], [19] and Weyler et al [1]. In our previous work [20], we perform joint stem detection for Rumex plants. We found that even for the human annotator it was challenging to identify the precise joint stem position, especially in settings where plants have reached advanced growing states, their structure increases in complexity meaning the joint stems can not be clearly identified.…”