Generating 3D Multispectral Point Clouds of Plants with Fusion of Snapshot Spectral and RGB-D Images

Abstract

Accurate and high-throughput plant phenotyping is important for accelerating crop breeding. Spectral imaging that can acquire both spectral and spatial information of plants related to structural, biochemical, and physiological traits becomes one of the popular phenotyping techniques. However, close-range spectral imaging of plants could be highly affected by the complex plant structure and illumination conditions, which becomes one of the main challenges for close-range plant phenotyping. In this study, we proposed a new method for generating high-quality plant 3-dimensional multispectral point clouds. Speeded-Up Robust Features and Demons was used for fusing depth and snapshot spectral images acquired at close range. A reflectance correction method for plant spectral images based on hemisphere references combined with artificial neural network was developed for eliminating the illumination effects. The proposed Speeded-Up Robust Features and Demons achieved an average structural similarity index measure of 0.931, outperforming the classic approaches with an average structural similarity index measure of 0.889 in RGB and snapshot spectral image registration. The distribution of digital number values of the references at different positions and orientations was simulated using artificial neural network with the determination coefficient ( R 2 ) of 0.962 and root mean squared error of 0.036. Compared with the ground truth measured by ASD spectrometer, the average root mean squared error of the reflectance spectra before and after reflectance correction at different leaf positions decreased by 78.0%. For the same leaf position, the average Euclidean distances between the multiview reflectance spectra decreased by 60.7%. Our results indicate that the proposed method achieves a good performance in generating plant 3-dimensional multispectral point clouds, which is promising for close-range plant phenotyping.

The Path to Smart Farming: Innovations and Opportunities in Precision Agriculture

“…High-throughput phenotyping has emerged as a promising approach to enhance precision agriculture by allowing the rapid and accurate measurement of plant traits [51] quantitatively and qualitatively [22,52]. Accurate and high-throughput plant phenotyping is important for accelerating crop breeding [52].…”

“…High-throughput phenotyping has emerged as a promising approach to enhance precision agriculture by allowing the rapid and accurate measurement of plant traits [51] quantitatively and qualitatively [22,52]. Accurate and high-throughput plant phenotyping is important for accelerating crop breeding [52]. This technique uses advanced technologies such as remote sensing [40,42], spectral imaging [41], and robotics [53] to collect large amounts of data on plant characteristics, such as growth rate, yield, disease resistance, and morphology [51,54,55].…”

“…A full range of visible and near-infrared hyperspectral data enables ML techniques such as LSR (least squares regression) to predict specific biochemical and physicochemical traits beyond simple vegetative indices [56]. ML-based precision agriculture systems have AI background [52,54], and therefore, when detecting diseases, pests, nutrient deficiency, and weeds, stressed responses are detected using high-quality images generated with UGV or UAV remote sensing, hyperspectral imaging, and satellite imaging to support high-throughput phenotyping [57]. Ultimately, high-throughput phenotyping has the potential to revolutionize agriculture by enabling more the precise, real-time, and efficient monitoring of farming practices that can improve crop productivity, reduce environmental impacts, and increase food security [54].…”

Fusion of Multimodal Imaging and 3D Digitization Using Photogrammetry

“…Multimodal data, in combination with shape, reveal new information about cultural heritage objects [23][24][25]. Drones capture multispectral data and topography of the terrain to derive the condition of field crops [26,27] and forest areas [28,29]. In industrial production, fusions of 3D and multimodal data have been investigated for the detection of contaminations [30] and human-robot interaction [31].…”

Simulation of Automatically Annotated Visible and Multi-/Hyperspectral Images Using the Helios 3D Plant and Radiative Transfer Modeling Framework

“…For instance, image annotation for wheat spikelet and ear counting is typically done manually [ 21 ], and manual selection of the canopy region of interest (ROI) is also required for many methods [ 7 ]. Additionally, the fusion of multimodal data also faces challenges when aligning images captured by different sensors due to their physically distinct viewpoints and resolutions [ 14 , 15 ].…”