Late fusion of multimodal deep neural networks for weeds classification

Trong, Vo Hoang; Yu, Gwang-Hyun; Vu, Dang Thanh; Kim, Jin‐Young

doi:10.1016/j.compag.2020.105506

Cited by 56 publications

(15 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, multimodal DL models are composed of multiple models, each trained using a single input type (e.g., rainfall, soil measurements, genetic data, hyperspectral imagery), or a single model trained on concatenated multimodal data ( Figure 2 ). The different modalities contribute to enrich the available features for model learning, contributing to an improved final prediction ( Baltrusaitis et al, 2019 ; Khaki et al, 2019 ; Gadiraju et al, 2020 ; Hoang Trong et al, 2020 ; Maimaitijiang et al, 2020 ). The use of multimodal models and other DL architectures, such as recurrent neural networks and graph neural networks, are still largely unexplored in genotype to phenotype predictions, but present a powerful alternative to traditional statistical methods.…”

Section: Integrating High Throughput Phenotyping Into Genotype To Phe...mentioning

confidence: 99%

Plant Genotype to Phenotype Prediction Using Machine Learning

et al. 2022

View full text Add to dashboard Cite

Genomic prediction tools support crop breeding based on statistical methods, such as the genomic best linear unbiased prediction (GBLUP). However, these tools are not designed to capture non-linear relationships within multi-dimensional datasets, or deal with high dimension datasets such as imagery collected by unmanned aerial vehicles. Machine learning (ML) algorithms have the potential to surpass the prediction accuracy of current tools used for genotype to phenotype prediction, due to their capacity to autonomously extract data features and represent their relationships at multiple levels of abstraction. This review addresses the challenges of applying statistical and machine learning methods for predicting phenotypic traits based on genetic markers, environment data, and imagery for crop breeding. We present the advantages and disadvantages of explainable model structures, discuss the potential of machine learning models for genotype to phenotype prediction in crop breeding, and the challenges, including the scarcity of high-quality datasets, inconsistent metadata annotation and the requirements of ML models.

show abstract

Section: Integrating High Throughput Phenotyping Into Genotype To Phe...mentioning

confidence: 99%

Plant Genotype to Phenotype Prediction Using Machine Learning

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Vo Hoang Trong et al [35] developed a novel classification approach via a voting method by using the late fusion of multimodal DNNs. The score vector used for voting is calculated by Bayesian conditional probability method or determining the priority, so that the score vector obtained from the classification model can get more weight in the final classification.…”

Section: Cnnmentioning

confidence: 99%

Image classification of forage grasses on Etuoke Banner using edge autoencoder network

et al. 2022

View full text Add to dashboard Cite

Automatically identifying the forage is the basis of intelligent fine breeding of cattle and sheep. In specific, it is a key step to study the relationship between the type and quantity of forage collected by cattle and sheep and their own growth, cashmere fineness, milk quality, meat quality and flavor, and so on. However, traditional method mainly rely on manual observation, which is time-consuming, laborious and inaccurate, and affects the normal grazing behavior of livestock. In this paper, the optimized Convolution Neural Network(CNN): edge autoencoder network(E-A-Net) algorithm is proposed to accurately identify the forage species, which provides the basis for ecological workers to carry out grassland evaluation, grassland management and precision feeding. We constructed the first forage grass dataset about Etuoke Banner. This dataset contains 3889 images in 22 categories. In the data preprocessing stage, the random cutout data enhancement is adopted to balance the original data, and the background is removed by employing threshold value-based image segmentation operation, in which the accuracy of herbage recognition in complex background is significantly improved. Moreover, in order to avoid the phenomenon of richer edge information disappearing in the process of multiple convolutions, a Sobel operator is utilized in this E-A-Net to extract the edge information of forage grasses. Information is integrated with the features extracted from the backbone network in multi-scale. Additionally, to avoid the localization of the whole information during the convolution process or alleviate the problem of the whole information disappearance, the pre-training autoencoder network is added to form a hard attention mechanism, which fuses the abstracted overall features of forage grasses with the features extracted from the backbone CNN. Compared with the basic CNN, E-A-Net alleviates the problem of edge information disappearing and overall feature disappearing with the deepening of network depth. Numerical simulations show that, compared with the benchmark VGG16, ResNet50 and EfficientNetB0, the f1 − score of the proposed method is improved by 1.6%, 2.8% and 3.7% respectively.

show abstract

“…We applied the metrics described in [23] to evaluate the performance of the conventional and YMufT strategies for training of a DNN model on an imbalanced dataset. Suppose the evaluation dataset D contains m images of c species.…”

Section: Performance Metricsmentioning

confidence: 99%

Yielding Multi-Fold Training Strategy for Image Classification of Imbalanced Weeds

et al. 2021

View full text Add to dashboard Cite

An imbalanced dataset is a significant challenge when training a deep neural network (DNN) model for deep learning problems, such as weeds classification. An imbalanced dataset may result in a model that behaves robustly on major classes and is overly sensitive to minor classes. This article proposes a yielding multi-fold training (YMufT) strategy to train a DNN model on an imbalanced dataset. This strategy reduces the bias in training through a min-class-max-bound procedure (MCMB), which divides samples in the training set into multiple folds. The model is consecutively trained on each one of these folds. In practice, we experiment with our proposed strategy on two small (PlantSeedlings, small PlantVillage) and two large (Chonnam National University (CNU), large PlantVillage) weeds datasets. With the same training configurations and approximate training steps used in conventional training methods, YMufT helps the DNN model to converge faster, thus requiring less training time. Despite a slight decrease in accuracy on the large dataset, YMufT increases the F1 score in the NASNet model to 0.9708 on the CNU dataset and 0.9928 when using the Mobilenet model training on the large PlantVillage dataset. YMufT shows outstanding performance in both accuracy and F1 score on small datasets, with values of (0.9981, 0.9970) using the Mobilenet model for training on small PlantVillage dataset and (0.9718, 0.9689) using Resnet to train on the PlantSeedlings dataset. Grad-CAM visualization shows that conventional training methods mainly concentrate on high-level features and may capture insignificant features. In contrast, YMufT guides the model to capture essential features on the leaf surface and properly localize the weeds targets.

show abstract

Late fusion of multimodal deep neural networks for weeds classification

Cited by 56 publications

References 11 publications

Plant Genotype to Phenotype Prediction Using Machine Learning

Plant Genotype to Phenotype Prediction Using Machine Learning

Image classification of forage grasses on Etuoke Banner using edge autoencoder network

Yielding Multi-Fold Training Strategy for Image Classification of Imbalanced Weeds

Contact Info

Product

Resources

About