Improving text mining in plant health domain with GAN and/or pre-trained language model

Abstract

The Bidirectional Encoder Representations from Transformers (BERT) architecture offers a cutting-edge approach to Natural Language Processing. It involves two steps: 1) pre-training a language model to extract contextualized features and 2) fine-tuning for specific downstream tasks. Although pre-trained language models (PLMs) have been successful in various text-mining applications, challenges remain, particularly in areas with limited labeled data such as plant health hazard detection from individuals' observations. To address this challenge, we propose to combine GAN-BERT, a model that extends the fine-tuning process with unlabeled data through a Generative Adversarial Network (GAN), with ChouBERT, a domain-specific PLM. Our results show that GAN-BERT outperforms traditional fine-tuning in multiple text classification tasks. In this paper, we examine the impact of further pre-training on the GAN-BERT model. We experiment with different hyper parameters to determine the best combination of models and fine-tuning parameters. Our findings suggest that the combination of GAN and ChouBERT can enhance the generalizability of the text classifier but may also lead to increased instability during training. Finally, we provide recommendations to mitigate these instabilities.

Multilingual Hate Speech Detection Using Semi-supervised Generative Adversarial Network

“…They implemented their model on several classification datasets, and they found that the performance of their semi-supervised model gets better every time increasing the size of labeled dataset. Moreover, Jiang et al [20] used CamemBERT, and ChouBERT in order to build GAN-BERT models. They also worked on examining varied losses over changing the number of labeled and unlabeled samples in the training French datasets in order to provide greater understanding into when and how to train GAN-BERT models for domain-specific document categorization.…”

DeepOCL: A deep neural network for Object Constraint Language generation from unrestricted nature language

“…The pre-trained language model has achieved great success in natural language processing (NLP). Inspired by this, a considerable amount of pre-trained models were proposed and applied for Software Engineering tasks, for example, services classification [16,17], code generation [18], code summarisation [19,20], code completion [21] and clone detection [15], achieving significant progress. In this paper, we adopt CodeT5 [15] as the base model.…”