LettuceTrack: Detection and tracking of lettuce for robotic precision spray in agriculture

Abstract

The precision spray of liquid fertilizer and pesticide to plants is an important task for agricultural robots in precision agriculture. By reducing the amount of chemicals being sprayed, it brings in a more economic and eco-friendly solution compared to conventional non-discriminated spray. The prerequisite of precision spray is to detect and track each plant. Conventional detection or segmentation methods detect all plants in the image captured under the robotic platform, without knowing the ID of the plant. To spray pesticides to each plant exactly once, tracking of every plant is needed in addition to detection. In this paper, we present LettuceTrack, a novel Multiple Object Tracking (MOT) method to simultaneously detect and track lettuces. When the ID of each plant is obtained from the tracking method, the robot knows whether a plant has been sprayed before therefore it will only spray the plant that has not been sprayed. The proposed method adopts YOLO-V5 for detection of the lettuces, and a novel plant feature extraction and data association algorithms are introduced to effectively track all plants. The proposed method can recover the ID of a plant even if the plant moves out of the field of view of camera before, for which existing Multiple Object Tracking (MOT) methods usually fail and assign a new plant ID. Experiments are conducted to show the effectiveness of the proposed method, and a comparison with four state-of-the-art Multiple Object Tracking (MOT) methods is shown to prove the superior performance of the proposed method in the lettuce tracking application and its limitations. Though the proposed method is tested with lettuce, it can be potentially applied to other vegetables such as broccoli or sugar beat.

Recent Advancements and Challenges of AIoT Application in Smart Agriculture: A Review

“…In addition to recognizing each plant, it is vital to track each plant to administer pesticides precisely once to each plant. The previous researcher proposed a multiple object tracking (MOT) technology that recognizes and tracks lettuce concurrently, only spraying plants that have never been treated before [ 85 ]. The approach leverages YOLO-V5 for identifying lettuce and includes plant feature extraction and data association algorithms to monitor all plants successfully.…”

Segmentation and tracking of vegetable plants by exploiting vegetable shape feature for precision spray of agricultural robots

“…In agricultural scenarios, tracking plants is generally difficult when conventional color and texture features are used (de Jong et al, 2022; Hu et al, 2022). Our previous work (Hu et al, 2022) presented a location information‐based feature extraction method based on the geometric relationship between the target plant and its neighboring plant.…”

“…In agricultural scenarios, tracking plants is generally difficult when conventional color and texture features are used (de Jong et al, 2022; Hu et al, 2022). Our previous work (Hu et al, 2022) presented a location information‐based feature extraction method based on the geometric relationship between the target plant and its neighboring plant. The method overcomes the challenging tracking problem of plants with similar appearance, but it requires the presence of multiple plants on an image to extract such location features.…”

“…Compared with MOT methods (Hu et al, 2022), which use bounding boxes to detect vegetables, MOTS provides an instance segmentation image mask, which can be used to infer unique characteristics and phenotypes of individual vegetable, such as their size and shape, to determine optimal dose of chemical spray. Current state-of-the-art (SOTA) MOTS methods are mostly designed for pedestrian or vehicle tracking, and have achieved remarkable results.…”

“…In this way, it forms the classic Multiple Object Tracking (MOT) (Bewley et al, 2016; Zhang et al, 2022) or Multiple Object Tracking and Segmentation (MOTS) problem (Gao et al, 2022; Voigtlaender et al, 2019). Compared with MOT methods (Hu et al, 2022), which use bounding boxes to detect vegetables, MOTS provides an instance segmentation image mask, which can be used to infer unique characteristics and phenotypes of individual vegetable, such as their size and shape, to determine optimal dose of chemical spray. Current state‐of‐the‐art (SOTA) MOTS methods are mostly designed for pedestrian or vehicle tracking, and have achieved remarkable results.…”