Multi-stage mixed rule learning approach for advancing performance of rule-based classification

Abstract

Rule learning is a special type of machine learning approaches, and its key advantage is the generation of interpretable models, which provides a transparent process of showing how an input is mapped to an output. Traditional rule learning algorithms are typically based on Boolean logic for inducing rule antecedents, which are very effective for training models on data sets that involve discrete attributes only. When continuous attributes are present in a data set, traditional rule learning approaches need to employ crisp intervals. However, in reality, problems usually show shades of grey, which motivated the development of fuzzy rule learning approaches by employing fuzzy intervals for handling continuous attributes. While a data set contains a large portion of discrete attributes or even no continuous attributes, fuzzy approaches cannot be used to learn rules effectively, leading to a drop in the performance. In this paper, a multi-stage approach of mixed rule learning is proposed, which involves strategic combination of both traditional and fuzzy approaches to handle effectively various types of attributes. We compare our proposed approach with existing algorithms of rule learning. Our experimental results show that our proposed approach leads to significant advances in the performance compared with the existing algorithms.

Heuristic creation of deep rule ensemble through iterative expansion of feature space

“…The results on classification accuracy for different methods are presented in Table 2. In particular, the proposed approach shows the top performance among all these existing approaches [9,28,31,37] in 17 out of the 20 cases, i.e., the proposed approach either outperforms all the other methods or performs the same as the best performing one(s) among the other methods. In columns 4-7 of Table 2, the four headers "PrismCTC1", "PrismCTC2", "PrismCTC3" and "PrismCTC4" represent that the PrismCTC algorithm is adopted with four different settings of the hyper-parameter named as "rule quality measure", where the four selected measures of rule quality are referred to as "confidence", "J-measure", "lift" and "leverage", respectively, which are explained in [31] in details.…”

“…After the 3-fold cross validation, it is also determined automatically whether the learning task continues by going for the next iteration i+1, i.e., the learning task would normally continue unless the learning performance (i.e., the classification accuracy measured using the 3-fold cross validation on the training set) is not advanced any further. The proposed approach is compared with a very recent approach MSMRL [28] as well as all the other methods (i.e., the C4.5 method [37], the Prism method [9] and the PrismCTC method [31]) that have been compared with the MSMRL method in [28]. The settings of the hyper-parameters for these existing methods (i.e., the C4.5 method, the Prism method, the PrismCTC method and the MSMRL method) are kept the same as the ones described in [28].…”

“…The proposed approach is compared with a very recent approach MSMRL [28] as well as all the other methods (i.e., the C4.5 method [37], the Prism method [9] and the PrismCTC method [31]) that have been compared with the MSMRL method in [28]. The settings of the hyper-parameters for these existing methods (i.e., the C4.5 method, the Prism method, the PrismCTC method and the MSMRL method) are kept the same as the ones described in [28]. The experiments on the 20 data sets are conducted through random splitting of data into training and test sets.…”

“…Another approach, which is referred to as "Multi-Stage Mixed Rule Learning" (MSMRL), has been proposed in [28] for creating rule ensembles that are gradually getting deeper, through learning from general structured data (i.e., the features in the data do not have spatial or sequential relationships). The MSMRL approach is designed to have a pre-defined number of iterations towards granually increasing the depth of learning rule ensembles, and also to involve multiple ways of diversity creation through diversification of features and heuristics for learning different rule-based classifiers at each iteration, while the C4.5 algorithm [37] and the Mixed Fuzzy Rule Formation algorithm [17] are adopted in a collaboative manner for effectively dealing with data sets that contain both discrete and continuous attributes (features), i.e., the C4.5 algorithm and the Mixed Fuzzy Rule Formation algorithm involve different heuristic ways for dealing with continuous attributes, and the C4.5 algorithm can also effectively deal with discrete attributes that can not be handled directly by the Mixed Fuzzy Rule Formation algorithm.…”

“…(4) The experimental results indicate that the proposed approach performs considerably better than some recent methods [28,31] of rule learning and ensemble creation on various data sets.…”