Multimodal Deep Learning via Late Fusion for Non-Destructive Papaya Fruit Maturity Classification

Due to the rapid emergence and evolution of AI applications, the utilization of smart imaging devices has increased significantly. Researchers have started using deep learning models, such as CNN, for image classification. Unlike the traditional models, which require a lot of features to perform well, CNN does not require any handcrafted features to perform well. It uses numerous filters, which extract required features from images automatically for classification. One of the issues in the horticulture industry is fruit classification, which requires an expert with a lot of experience. To overcome this issue an automated system is required which can classify different types of fruits without the need for any human effort. In this study, a dataset of a total of 26,149 images of 40 different types of fruits was used for experimentation. The training and test set were randomly recreated and divided into the ratio of 3:1. The experiment introduces a customized head of five different layers into MobileNetV2 architecture. The classification layer of the MobileNetV2 model is replaced by the customized head, which produced the modified version of MobileNetV2 called TL-MobileNetV2. In addition, transfer learning is used to retain the pre-trained model. TL-MobileNetV2 achieves an accuracy of 99%, which is 3% higher than MobileNetV2, and the equal error rate of TL-MobileNetV2 is just 1%. Compared to AlexNet, VGG16, InceptionV3, and ResNet, the accuracy is better by 8, 11, 6, and 10%, respectively. Furthermore, the TL-MobileNetV2 model obtained 99% precision, 99% for recall, and a 99% F1-score. It can be concluded that transfer learning plays a big part in achieving better results, and the dropout technique helps to reduce the overfitting in transfer learning.

show abstract

Section: Introductionmentioning

confidence: 99%

“…There have been some studies that focused on the quality of the fruits, such as [20][21][22][23]. Garillos-Manliguez et al [20] proposed a model for the estimation of the maturity of papaya fruit. The unique thing about this model is that it is trained on hyperspectral and visible-light images, unlike other models.…”

Section: Introductionmentioning

confidence: 99%

Fruit Image Classification Model Based on MobileNetV2 with Deep Transfer Learning Technique

Gulzar

2023

Sustainability

134

View full text Add to dashboard Cite

show abstract

“…Currently, multimodal fusion techniques have been used in a variety of fields, such as medical diagnosis [ 29 ], emotion recognition [ 30 ], education [ 31 ], industrial fault diagnosis [ 32 ], and autonomous driving [ 33 ]. A number of reports have applied multimodal fusion technology to agriculture-related research [ 34 , 35 , 36 , 37 , 38 ]. The application of multimodal fusion technology in agricultural technology, especially in the assessment of fruit and vegetable maturity, not only improves the accuracy and efficiency of the assessment but also provides a new perspective for agricultural production and quality assessment supply chain management.…”

Section: Introductionmentioning

confidence: 99%

Development of Multimodal Fusion Technology for Tomato Maturity Assessment

Liu,

Wei,

Yoon

et al. 2024

Sensors

View full text Add to dashboard Cite

The maturity of fruits and vegetables such as tomatoes significantly impacts indicators of their quality, such as taste, nutritional value, and shelf life, making maturity determination vital in agricultural production and the food processing industry. Tomatoes mature from the inside out, leading to an uneven ripening process inside and outside, and these situations make it very challenging to judge their maturity with the help of a single modality. In this paper, we propose a deep learning-assisted multimodal data fusion technique combining color imaging, spectroscopy, and haptic sensing for the maturity assessment of tomatoes. The method uses feature fusion to integrate feature information from images, near-infrared spectra, and haptic modalities into a unified feature set and then classifies the maturity of tomatoes through deep learning. Each modality independently extracts features, capturing the tomatoes’ exterior color from color images, internal and surface spectral features linked to chemical compositions in the visible and near-infrared spectra (350 nm to 1100 nm), and physical firmness using haptic sensing. By combining preprocessed and extracted features from multiple modalities, data fusion creates a comprehensive representation of information from all three modalities using an eigenvector in an eigenspace suitable for tomato maturity assessment. Then, a fully connected neural network is constructed to process these fused data. This neural network model achieves 99.4% accuracy in tomato maturity classification, surpassing single-modal methods (color imaging: 94.2%; spectroscopy: 87.8%; haptics: 87.2%). For internal and external maturity unevenness, the classification accuracy reaches 94.4%, demonstrating effective results. A comparative analysis of performance between multimodal fusion and single-modal methods validates the stability and applicability of the multimodal fusion technique. These findings demonstrate the key benefits of multimodal fusion in terms of improving the accuracy of tomato ripening classification and provide a strong theoretical and practical basis for applying multimodal fusion technology to classify the quality and maturity of other fruits and vegetables. Utilizing deep learning (a fully connected neural network) for processing multimodal data provides a new and efficient non-destructive approach for the massive classification of agricultural and food products.

show abstract

“…With the development of deep learning, the extraction of image features has been automated, and the construction of classification models has become increasingly simple [ 34 , 35 , 36 ]. The authors of [ 37 ] constructed a maturity detection model for tomatoes based on an improved DenseNet, with a detection rate of up to 91.26%.…”

Section: Introductionmentioning

confidence: 99%

Tomato Maturity Detection and Counting Model Based on MHSA-YOLOv8

Zheng

et al. 2023

Sensors

View full text Add to dashboard Cite

The online automated maturity grading and counting of tomato fruits has a certain promoting effect on digital supervision of fruit growth status and unmanned precision operations during the planting process. The traditional grading and counting of tomato fruit maturity is mostly done manually, which is time-consuming and laborious work, and its precision depends on the accuracy of human eye observation. The combination of artificial intelligence and machine vision has to some extent solved this problem. In this work, firstly, a digital camera is used to obtain tomato fruit image datasets, taking into account factors such as occlusion and external light interference. Secondly, based on the tomato maturity grading task requirements, the MHSA attention mechanism is adopted to improve YOLOv8’s backbone to enhance the network’s ability to extract diverse features. The Precision, Recall, F1-score, and mAP50 of the tomato fruit maturity grading model constructed based on MHSA-YOLOv8 were 0.806, 0.807, 0.806, and 0.864, respectively, which improved the performance of the model with a slight increase in model size. Finally, thanks to the excellent performance of MHSA-YOLOv8, the Precision, Recall, F1-score, and mAP50 of the constructed counting models were 0.990, 0.960, 0.975, and 0.916, respectively. The tomato maturity grading and counting model constructed in this study is not only suitable for online detection but also for offline detection, which greatly helps to improve the harvesting and grading efficiency of tomato growers. The main innovations of this study are summarized as follows: (1) a tomato maturity grading and counting dataset collected from actual production scenarios was constructed; (2) considering the complexity of the environment, this study proposes a new object detection method, MHSA-YOLOv8, and constructs tomato maturity grading models and counting models, respectively; (3) the models constructed in this study are not only suitable for online grading and counting but also for offline grading and counting.

show abstract

Multimodal Deep Learning via Late Fusion for Non-Destructive Papaya Fruit Maturity Classification

Cited by 4 publications

References 17 publications

Fruit Image Classification Model Based on MobileNetV2 with Deep Transfer Learning Technique

Fruit Image Classification Model Based on MobileNetV2 with Deep Transfer Learning Technique

Development of Multimodal Fusion Technology for Tomato Maturity Assessment

Tomato Maturity Detection and Counting Model Based on MHSA-YOLOv8

Contact Info

Product

Resources

About