Simulation-Based Assessment of Classification Methods: Statistical Models vs. Machine Learning Algorithms

Abstract

Current studies evaluated the effectiveness of categorization techniques primarily using real datasets with unreported or unknown statistical features. This simulation-based study aims to compare the performance of statistical models (logistic regression, probit regression, and discriminant analysis) with machine learning algorithms (support vector machines, classification and regression trees, and k-nearest neighbors) to comprehensively understand their suitability for classification tasks. Although simulated datasets are used to control their statistical characteristics, the Pima Indian Diabetes real dataset is used to verify the study findings. The outcomes of this study have the potential to guide practitioners and researchers in selecting the most appropriate modeling technique for their specific needs, ultimately enhancing the accuracy and reliability of classification outcomes across various domains. The results revealed that the two statistical models -probit and logit-outperformed in most simulation scenarios. Markedly, the well-grounded, theory-based models of the logit regression and the probit regression models yielded the most accurate predictions in 78.5% and 83.6% of the simulated scenarios, respectively. Interestingly, the performance of the probit model was the best when the binary response variable was balanced (τ=0.50) and when it was too imbalanced (τ=0.90). Notably, the resulting performance metrics of the real dataset refer to the logit, followed by the probit, being the best-predicting models, which resembles the outcome of the simulation study.in the case of binary response variables. Although different categorical response models exist, the most commonly applied are the logistic regression, the probit regression, and the Discriminant Analysis (DA).The term ML was first introduced by Arthur Samuel in 1959 (Arthur, 1959). ML is the field of study that trains computers/systems to operate independently and improve with experience. Accordingly, ML algorithms construct a model based on sample data-training data-to make predictions or decisions. Furthermore, ML utilizes notions from various disciplines: statistics, mathematics, philosophy, computational complexity, and artificial intelligence. Markedly, interest in applying contemporary ML techniques as alternatives to statistical methods is widely increasing (Lynam et al., 2020). For that, colossal improvement has been achieved by ML methods concerning the simple binary discrimination problem that qualitative response models can target.Further, it was claimed that the successful use of ML in several fields indicates promising applications in other fields. However, the advantages and superiority of ML-based classification methods compared with more traditional statistical ones need to be assessed, validated, and verified in all fields of application (Côté et al., 2022). With that in mind, such alternative ML algorithms include Decision Trees (DTs), Support Vector Machines (SVMs), K-Nearest Neighbors (KNN), Random Forest (RF), Gaussian Process (G...