WEC: Weighted Ensemble of Text Classifiers

Abstract

Text classification is one of the most important tasks in the field of Natural Language Processing. There are many approaches that focus on two main aspects: generating an effective representation; and selecting and refining algorithms to build the classification model. Traditional machine learning methods represent documents in vector space using features such as term frequencies, which have limitations in handling the order and semantics of words. Meanwhile, although achieving many successes, deep learning classifiers require substantial resources in terms of labelled data and computational complexity. In this work, a weighted ensemble of classifiers (WEC) is introduced to address the text classification problem. Instead of using majority vote as the combining method, we propose to associate each classifier's prediction with a different weight when combining classifiers. The optimal weights are obtained by minimising a loss function on the training data with the Particle Swarm Optimisation algorithm. We conducted experiments on 5 popular datasets and report classification performance of algorithms with classification accuracy and macro F1 score. WEC was run with several different combinations of traditional machine learning and deep learning classifiers to show its flexibility and robustness. Experimental results confirm the advantage of WEC, especially on smaller datasets.

Detection of Cyberbullying Through BERT and Weighted Ensemble of Classifiers

“…As in [35], the main concept of PSO is a particle involving two components: a position vector which refers to a potential solution (candidate) for the optimisation problem and a velocity vector. A set of positions were initialised as follows: w ( i 0) (i = 1, .…”

Detection of Cyberbullying Through BERT and Weighted Ensemble of Classifiers

“…Combining different high performing classifiers into an ensemble proved to be an effective performance boost in a recent study [35]. The researchers found that an ensemble of models using different data representations balanced the strengths and weaknesses of each data representation.…”

“…As in [35], the equations 7 and 8 will be used in the PSO objective function to generate a macro F1 score from the collective predictions vs the true predictions. Since PSO is only set up to find minima and not maxima, the F1 score will have to be inverted (1/F1 score) before the value is returned.…”

“…Once all models have been tested and fine tuned, the best performing models will be selected to be included in a custom ensemble method based on the one described in Uphadhyay et al (2020) [35] where Particle Swarm Optimisation (PSO) will be used to find optimal weightings to be applied to the prediction probabilities for each classifier. The hope here is that different data representations will iron out the weaknesses of the best model when it is unsure on a prediction.…”

Ensemble-Based Short Text Similarity: An Easy Approach for Multilingual Datasets Using Transformers and WordNet in Real-World Scenarios

“…There are different voting schemes, such as majority voting, where the predicted class with the highest number of votes is selected, and weighted voting, where the models' predictions are weighted based on their performance or expertise. Weight-based and majority vote-based models play an important role in image classification, medicine, and manufacturing, enabling the prediction of failures and providing predictive support [30][31][32][33][34]. We used both methods in different phases to identify the more accurate results.…”