Implementing spatio-temporal 3D-convolution neural networks and UAV time series imagery to better predict lodging damage in sorghum.

Abstract

Unmanned aerial vehicle (UAV)-based remote sensing is gaining momentum in a variety of agricultural and environmental applications. Very high-resolution remote sensing image sets collected repeatedly across a crop growing season are becoming increasingly common. Analytical methods able to learn from both spatial and time dimensions of the data may allow improved estimation of crop traits, as well as the effects of genetics and the environment upon them. Multispectral and geometric time series imagery was collected by UAV on 11 dates, along with ground-truth data, in a field trial of 866 genetically diverse biomass sorghum accessions. We compared the performance of Convolution Neural Network (CNN) architectures that used image data from single dates (two spatial dimensions, 2D) versus multiple dates (two spatial dimensions + temporal dimension, 3D) to estimate lodging detection and severity. Lodging was detected with 3D-CNN analysis of time-series imagery with 0.88 accuracy, 0.92 precision, and 0.83 recall. This outperformed the best 2D-CNN on a single date with 0.85 accuracy, 0.84 precision, and 0.76 recall. Variation in lodging severity was estimated by the best 3D-CNN analysis with 9.4% mean absolute error (MAE), 11.9% root mean square error (RMSE), and goodness-of-fit (R2) of 0.76. This was a significant improvement over the best 2D-CNN analysis with 11.84% MAE, 14.91% RMSE, and 0.63 R2. Success of the improved 3D-CNN analysis approach depended on inclusion of before and after data i.e. images collected on dates before and after the lodging event. Integration of geometric and spectral features with 3D-CNN architecture was also key to improved assessment of lodging severity, which is an important and difficult to assess phenomenon in bioenergy feedstocks such as biomass sorghum. This demonstrates that spatio-temporal CNN architectures based on UAV time series imagery have significant potential to enhance plant phenotyping capabilities in crop breeding and precision agriculture applications.