A Comparison Study on Rule Extraction from Neural Network Ensembles, Boosted Shallow Trees, and SVMs

Abstract

One way to make the knowledge stored in an artificial neural network more intelligible is to extract symbolic rules. However, producing rules from Multilayer Perceptrons (MLPs) is an NP-hard problem. Many techniques have been introduced to generate rules from single neural networks, but very few were proposed for ensembles. Moreover, experiments were rarely assessed by 10-fold cross-validation trials. In this work, based on the Discretized Interpretable Multilayer Perceptron (DIMLP), experiments were performed on 10 repetitions of stratified 10-fold cross-validation trials over 25 binary classification problems. The DIMLP architecture allowed us to produce rules from DIMLP ensembles, boosted shallow trees (BSTs), and Support Vector Machines (SVM). The complexity of rulesets was measured with the average number of generated rules and average number of antecedents per rule. From the 25 used classification problems, the most complex rulesets were generated from BSTs trained by "gentle boosting" and "real boosting." Moreover, we clearly observed that the less complex the rules were, the better their fidelity was. In fact, rules generated from decision stumps trained by modest boosting were, for almost all the 25 datasets, the simplest with the highest fidelity. Finally, in terms of average predictive accuracy and average ruleset complexity, the comparison of some of our results to those reported in the literature proved to be competitive.

Logic-Based Technologies for Intelligent Systems: State of the Art and Perspectives

“…Several works in recent decades proposed to extract symbolic knowledge from numeric models. As witnessed by several surveys [158,[171][172][173] and works on the topic [174][175][176][177][178][179][180][181][182][183][184]-some of which are from the 80s or the 90s-the potential of symbolic knowledge extraction is well understood, although without hype.…”

Copying Machine Learning Classifiers

“…An important degree of freedom in distillation is the transfer set used to train the simpler model. Traditionally, knowledge transfer has been treated as a standard learning process, where the training data are relabelled and extended to learn an alternative model [26]. Most papers use the same set to train teacher and student, either in its raw form [26], [13], [27] or enriched with additional synthetic data [28], [11], [17].…”

“…Traditionally, knowledge transfer has been treated as a standard learning process, where the training data are relabelled and extended to learn an alternative model [26]. Most papers use the same set to train teacher and student, either in its raw form [26], [13], [27] or enriched with additional synthetic data [28], [11], [17]. Besides, researchers have also studied cases where teachers and students faced with the same task have different access to the training data [29].…”

Combination of Active Learning and Semi-Supervised Learning under a Self-Training Scheme

“…The plethora of experiments carried out, also play a major role in the validation of the proposed algorithm. The proposed method is examined through a number of different individual base learners, where the ensemble learning technique is also explored as the aggregated models tend to produce more accurate predictions and are commonly used in today’s applications [ 9 , 10 ].…”