Crop Agnostic Monitoring Driven by Deep Learning

Abstract

Farmers require diverse and complex information to make agronomical decisions about crop management including intervention tasks. Generally, this information is gathered by farmers traversing their fields or glasshouses which is often a time consuming and potentially expensive process. In recent years, robotic platforms have gained significant traction due to advances in artificial intelligence. However, these platforms are usually tied to one setting (such as arable farmland), or algorithms are designed for a single platform. This creates a significant gap between available technology and farmer requirements. We propose a novel field agnostic monitoring technique that is able to operate on two different robots, in arable farmland or a glasshouse (horticultural setting). Instance segmentation forms the backbone of this approach from which object location and class, object area, and yield information can be obtained. In arable farmland, our segmentation network is able to estimate crop and weed at a species level and in a glasshouse we are able to estimate the sweet pepper and their ripeness. For yield information, we introduce a novel matching criterion that removes the pixel-wise constraints of previous versions. This approach is able to accurately estimate the number of fruit (sweet pepper) in a glasshouse with a normalized absolute error of 4.7% and an R2 of 0.901 with the visual ground truth. When applied to cluttered arable farmland scenes it improves on the prior approach by 50%. Finally, a qualitative analysis shows the validity of this agnostic monitoring algorithm by supplying decision enabling information to the farmer such as the impact of a low level weeding intervention scheme.

Development and evaluation of automated localization and reconstruction of all fruits on tomato plants in a greenhouse based on multi-view perception and 3D multi-object tracking

“…They were able to count strawberries with a MAPE of 3%; although their conditions were more favorable with fewer occlusions and a larger distance between individual fruits. In similar environment conditions as ours, Halstead et al (2021) developed and evaluated another similar tracking system to count bell peppers in individual plants, and achieved a MAPE of 4.1%. Their data also contained clutter and de-leafed areas of plants, but, since bell peppers do not grow in cluster like tomatoes, the distance between individual fruits is more favorable toward tracking.…”

“…MOT methods have been recently used in agro-food robotic systems (Halstead et al, 2018;Kirk et al, 2021;Smitt et al, 2021;Halstead et al, 2021). Nevertheless, their objectives were to perform specific tasks like plant phenotyping and monitoring, and were not aiming for a generic representation to improve the capabilities of an autonomous robotic system in tasks like harvesting and crop maintenance.…”

“…They made use of the 3D data obtained from the camera mounted in the robotic system to re-project the bounding boxes of tracked objects onto the following frames, then the tracking was performed by the previously mentioned IoU-based system. Halstead et al (2021) further extended the work of Smitt et al (2021) to different cropping systems, like arable farmland, and evaluated the performance on sugar beet plant counting. Kirk et al (2021) also developed an IoU-based tracking system assisted by deep learning, inspired by the work done by Wojke et al (2017).…”

“…The above mention studies of Halstead et al (2018); Kirk et al (2021); Smitt et al (2021); Halstead et al (2021) have the use of an autonomous systems with only one DoF that moves and collects data in a line parallel to the plant row or soil in common. This could be enough for monitoring applications in cropping systems where the objects of interest grow relatively wide spaced, as is the case for strawberries or already pruned bell peppers.…”

Explicitly Incorporating Spatial Information to Recurrent Networks for Agriculture

“…From the early work of Nuske et al [5] and even the more recent deep convolutional neural network (DCNN) approach of Sa et al [6]. It has only been recent work that highlighted the potential of spatial and temporal cues as a rich source of information to improve the outputs and classification results of DCNNs [7]- [9] applied to agriculture. Despite these advances, it remains an open question about how best to combine the spatial and temporal Fig.…”