Multi-Modal Detection and Mapping of Static and Dynamic Obstacles in Agriculture for Process Evaluation

Abstract

Today, agricultural vehicles are available that can automatically perform tasks such as weed detection and spraying, mowing, and sowing while being steered automatically. However, for such systems to be fully autonomous and self-driven, not only their specific agricultural tasks must be automated. An accurate and robust perception system automatically detecting and avoiding all obstacles must also be realized to ensure safety of humans, animals, and other surroundings. In this paper, we present a multi-modal obstacle and environment detection and recognition approach for process evaluation in agricultural fields. The proposed pipeline detects and maps static and dynamic obstacles globally, while providing process-relevant information along the traversed trajectory. Detection algorithms are introduced for a variety of sensor technologies, including range sensors (lidar and radar) and cameras (stereo and thermal). Detection information is mapped globally into semantical occupancy grid maps and fused across all sensors with late fusion, resulting in accurate traversability assessment and semantical mapping of process-relevant categories (e.g., crop, ground, and obstacles). Finally, a decoding step uses a Hidden Markov model to extract relevant process-specific parameters along the trajectory of the vehicle, thus informing a potential control system of unexpected structures in the planned path. The method is evaluated on a public dataset for multimodal obstacle detection in agricultural fields. Results show that a combination of multiple sensor modalities increases detection performance and that different fusion strategies must be applied between algorithms detecting similar and dissimilar classes.

Development of a sweet pepper harvesting robot

“…Due to the complex problem, most R&D on robotic harvesting focuses on a single aspect of the robotic system, for example, detection (Halstead, McCool, Denman, Perez, & Fookes, 2018;Kamilaris & Prenafeta-Boldú, 2018;Kapach, Barnea, Mairon, Edan, & Ben-Shahar, 2012;Vitzrabin & Edan, 2016a, 2016bZemmour, Kurtser, & Edan, 2019;Zhao, Gong, Huang, & Liu, 2016), manipulation and gripping (Bulanon & Kataoka, 2010;Eizicovits & Berman, 2014;Eizicovits, van Tuijl, Berman, & Edan, 2016;Rodríguez, Moreno, Sánchez, & Berenguel, 2013;Tian, Zhou, & Gu, 2018), and motion/task planning (Barth, IJsselmuiden, Hemming, & Van Henten, 2016; Korthals et al, 2018;Li & Qi, 2018;Liu, ElGeneidy, Pearson, Huda, & Neumann, 2018;.…”

Lidar-based Obstacle Detection and Recognition for Autonomous Agricultural Vehicles

“…From these, precision, recall, and F 1 scores (harmonic mean of precision and recall) were derived along with the entropy H describing information gain. More details on the metrics and how the evaluation was carried out specifically are available in Paper 7 (Korthals et al, 2018). In the following, each of the three scenarios is presented individually.…”

“…The dataset was used in Paper 7 (Korthals et al, 2018) for multi-modal detection and mapping of static and dynamic obstacles, and in Paper 9 (Kragh et al, 2018) for multimodal semantic segmentation in 3D.…”

“…This reduces the number of inputs and handles potential positional offsets between layers by clustering grid cells with similar predictions. The model and training procedures are further described in Paper 7 (Korthals et al, 2018).…”

“…Therefore, cell-wise evaluation was infeasible. Instead, algorithm predictions were clustered, and TP, FP, and FN figures were calculated by comparing detected clusters with ground truth points for each timestamp as described in Paper 7 (Korthals et al, 2018).…”