Two-Stream Dense Feature Fusion Network Based on RGB-D Data for the Real-Time Prediction of Weed Aboveground Fresh Weight in a Field Environment

Abstract

The aboveground fresh weight of weeds is an important indicator that reflects their biomass and physiological activity and directly affects the criteria for determining the amount of herbicides to apply. In precision agriculture, the development of models that can accurately locate weeds and predict their fresh weight can provide visual support for accurate, variable herbicide application in real time. In this work, we develop a two-stream dense feature fusion convolutional network model based on RGB-D data for the real-time prediction of the fresh weight of weeds. A data collection method is developed for the compilation and production of RGB-D data sets. The acquired images undergo data enhancement, and a depth transformation data enhancement method suitable for depth data is proposed. The main idea behind the approach in this study is to use the YOLO-V4 model to locate weeds and use the two-stream dense feature fusion network to predict their aboveground fresh weight. In the two-stream dense feature fusion network, DenseNet and NiN methods are used to construct a Dense-NiN-Block structure for deep feature extraction and fusion. The Dense-NiN-Block module was embedded in five convolutional neural networks for comparison, and the best results were achieved with DenseNet201. The test results show that the predictive ability of the convolutional network using RGB-D as the input is better than that of the network using RGB as the input without the Dense-NiN-Block module. The mAP of the proposed network is 75.34% (IoU value of 0.5), the IoU is 86.36%, the detection speed of the fastest model with a RTX2080Ti NVIDIA graphics card is 17.8 fps, and the average relative error is approximately 4%. The model proposed in this paper can provide visual technical support for precise, variable herbicide application. The model can also provide a reference method for the non-destructive prediction of crop fresh weight in the field and can contribute to crop breeding and genetic improvement.

Estimation of Greenhouse Lettuce Growth Indices Based on a Two-Stage CNN Using RGB-D Images

“…For the depth images, the crop area was decreased to 256 × 256 pixels to minimize the effect of the background. The k-nearest neighbor method was applied to fill the pixels for which the depth image data were not acquired, and the average value of nine pixels was used [ 27 , 40 ].…”

“…In addition, 3D data, such as RGB-D images, may provide more information on vertical changes in leaves and stems [ 23 , 24 , 25 , 26 ]. Quan et al developed a novel two-stream CNN model based on RGB-D images to estimate the fresh weight of aboveground weeds in a field on a high-end graphics processing unit (GPU) (2080Ti, NVIDIA, Santa Clara, CA, USA) [ 27 ]. The two-stream CNN model used a multi-input single-output (MISO) structure and dense network in the network blocks.…”

A Review of Landcover Classification with Very-High Resolution Remotely Sensed Optical Images—Analysis Unit, Model Scalability and Transferability

“…For example, long short-term memory (LSTM) and CNN were both used to extract features from social sensing signature data, which were concatenated with features extracted with CNN from spectral images to classify urban region functions [171]. The feature fusion may also be performed in intermediate layers in addition to the last layer [172][173][174][175], which is indicated by the dashed arrow lines in Figure 4b. In addition to concatenation, features can also be fused by maximum extraction operation, i.e., for each position in the feature vector, selecting the maximum values among the features extracted across all the data sources [175].…”

Recognition of Manual Welding Positions from Depth Hole Image Remotely Sensed by RGB-D Camera

“…Many studies have been conducted on body tracking and motion analysis using depth images [20][21][22][23]. There are two methods for creating depth images: extracting features from two-dimensional images and inferring depth through learning [24][25][26][27] or shooting with a 3D depth camera [28][29][30]. The former method has disadvantages in that an additional process is required to extract and learn features of an image, it takes a lot of time, and the accuracy is low.…”