ChouBERT: Pre-training French Language Model for Crowdsensing with Tweets in Phytosanitary Context

Abstract

To fulfil the increasing need for food of the growing population and face climate change, modern technologies have been applied to improve different farming processes. One important application scenario is to detect and measure natural hazards using sensors and data analysis techniques. Crowdsensing is a sensing paradigm that empowers ordinary people to contribute with data their sensor-enhanced mobile devices gather or generate. In this paper, we propose to use Twitter as an open crowdsensing platform for acquiring farmers knowledge. We proved this concept by applying pre-trained language models to detect individual's observation from tweets for pest monitoring.

Improving text mining in plant health domain with GAN and/or pre-trained language model

“…Therefore, the main challenge of detecting natural hazards from textual contents on social media is to identify unseen risks with low resources for training. We reuse the labeled tweets produced by ChouBERT (Jiang et al, 2022 ), tweets about corn borer, barley yellow dwarf virus (BYDV) and corvids for training and validation, and tweets about unseen and polysemous terms such as “taupin” (wireworm in English) for testing the generalizability of the classifier. Since the binary cross entropy loss adopted by the discriminator of GAN-BERT favors the majority class when data are unbalanced, for the different training experiments, we sampled ChouBERT's training data to 16, 32, 64, 128, 256, and 512 subsets, each subset having equal number of observations and non-observations.…”

“…Following the study of Jiang et al ( 2022 ), the ChouBERT models 6 are further-pre-trained CamemBERT-base models over French Plant Health Bulletins and Tweets and the ChouBERT pre-trained for 16 epochs (denoted as ChouBERT-16) and for 32 epochs (denoted as ChouBERT-32) are the most efficient in finding observations about plant health issues. Thus, in this study, we combine GAN-BERT settings with CamemBERT, ChouBERT-16, and ChouBERT-32.…”

“…When applying the PLM-only classification to other datasets in other domains, we might need to find an optimal threshold depending on the real needs for precision or recall. Based on the results presented in the study of Jiang et al ( 2022 ), we fixed the learning rate to 2e −5 , the maximum sequence length to 128, and fit the classifier for 10 epochs. We set the batch size to ( training _ data _ size /8) to have the same steps for the different training data sizes.…”

“…Still, the extraction of useful plant health information from social media poses some challenges, including lack of context, irrelevancy, homographs, homophones, homonyms, slangs, and colloquialisms. In an earlier study, we developed ChouBERT (Jiang et al, 2022 ) to detect farmers' observations from tweets for pest monitoring. ChouBERT takes a pre-trained CamemBERT (Martin et al, 2020 ) model and further pre-trains it on a plant health domain corpus in French to improve the generalizability of plant health hazards detection on Twitter.…”

“…Among the French varieties of BERT, CamemBERT (Martin et al, 2020 ) is a model based on the same architecture as BERT but trained on a French corpus with MLM only. ChouBERT (Jiang et al, 2022 ) takes a pre-trained CamemBERT-base checkpoint and further pre-trains it with MLM over a corpus in French in the plant health domain to improve performance in detecting plant health issues from short texts, particularly, from Twitter.…”