Efficient lazy elimination for averaged one-dependence estimators

Abstract

Semi-naive Bayesian classifiers seek to retain the numerous strengths of naive Bayes while reducing error by relaxing the attribute independence assumption. Backwards Sequential Elimination (BSE) is a wrapper technique for attribute elimination that has proved effective at this task. We explore a new technique, Lazy Elimination (LE), which eliminates highly related attribute-values at classification time without the computational overheads inherent in wrapper techniques. We analyze the effect of LE and BSE on a state-of-the-art semi-naive Bayesian algorithm Averaged One-Dependence Estimators (AODE). Our experiments show that LE significantly reduces bias and error without undue computation, while BSE significantly reduces bias but not error, with high training time complexity. In the context of AODE, LE has a significant advantage over BSE in both computational efficiency and error.

To Select or To Weigh: A Comparative Study of Model Selection and Model Weighing for SPODE Ensembles

“…Only selected SPODEs will be included in the ensemble. Previous research has suggested that cross validation, forward sequential addition and lazy elimination are more effective than alternative selection methods for SPODEs [22,23].…”

Finding the Right Family: Parent and Child Selection for Averaged One-Dependence Estimators

“…When there are many more attributes than those that participate in a particular inter-dependency, the majority of ODEs will not factor out the inter-dependency, and hence it is credible that deleting one of the attributes should be beneficial. Why then have previous attempts [15,19] to apply attribute-selection to AODE proved unfruitful? One difference between applying attribute selection in NB compared to AODE may be the greater complexity of an AODE model, resulting in greater variance in estimates of performance as the model is manipulated through attribute elimination and hence reduced reliability in these estimates.…”

“…The second approach, called child elimination (CE), deletes attribute indexes from c, effectively deleting an attribute from within every ODE at each step. Parent and child elimination (P∧CE) [15] at each step deletes the same value from both p and c, thus eliminating it from use in any role in the classifier. Parent or child elimination (P∨CE) performs any one of the other types of attribute eliminations in each iteration, selecting the option that best reduces error.…”

“…Semi-naive Bayesian techniques further improve naive Bayes' accuracy by relaxing its assumption that the attributes are conditionally independent [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]. One approach to weakening this assumption is to use an x-dependence classifier [7], in which each attribute depends upon the class and at most x other attributes.…”

“…When applied to naive Bayes, FSS and BSE have proved to be beneficial in domains with highly correlated attributes. It is therefore surprising that two attempts to apply these approaches to AODE have proved ineffective [15,19]. Where the training time overheads of attribute selection are not a major concern, attribute selection has the potential of two beneficial effects, both of improving accuracy and also of reducing test time due to the need to process fewer attributes.…”