The Effect of Resampling on Classifier Performance: an Empirical Study

Abstract

An imbalanced class on a dataset is a common classification problem. The effect of using imbalanced class datasets can cause a decrease in the performance of the classifier. Resampling is one of the solutions to this problem. This study used 100 datasets from 3 websites: UCI Machine Learning, Kaggle, and OpenML. Each dataset will go through 3 processing stages: the resampling process, the classification process, and the significance testing process between performance evaluation values of the combination of classifier and the resampling using paired t-test. The resampling used in the process is Random Undersampling, Random Oversampling, and SMOTE. The classifier used in the classification process is Naïve Bayes Classifier, Decision Tree, and Neural Network. The classification results in accuracy, precision, recall, and f-measure values are tested using paired t-tests to determine the significance of the classifier's performance from datasets that were not resampled and those that had applied the resampling. The paired t-test is also used to find a combination between the classifier and the resampling that gives significant results. This study obtained two results. The first result is that resampling on imbalanced class datasets can substantially affect the classifier's performance more than the classifier's performance from datasets that are not applied the resampling technique. The second result is that combining the Neural Network Algorithm without the resampling provides significance based on the accuracy value. Combining the Neural Network Algorithm with the SMOTE technique provides significant performance based on the amount of precision, recall, and f-measure.

WELDONNet: Weighted Ensemble DenseNet121, MobileNetV2, and NASNetMobile to improve skin lesion diseases classification

“…In addition, random down-sampling was implemented for Melanocytic Nevus and random up-sampling for the rest. This technique was evident to improve classi cation performance, according to research conducted by [29]. Figure 1 illustrates the image resize results, and Fig.…”

“…However, this method sometimes generates less-favorable results. More recently [29], [30], the proposed random up-sampling and down-sampling techniques by considering unbalanced datasets has been implemented to achieve a more improved classi cation performance. The last challenges in designing skin lesion CAD systems that employ handcrafted and automatic-based deep learning CNN algorithms are over tting the model requiring high cost in terms of time complexity.…”

Exploring Visitor Sentiments: A Study of Nusantara Temple Reviews on TripAdvisor Using Machine Learning

“…This disparity might impact the model's performance or the upcoming analysis. The preprocessing stage involves a resampling method utilizing the synthetic minority oversampling technique (SMOTE) [19] to rectify the data quantity imbalance.…”