Feature Selection based on the Local Lift Dependence Scale

Abstract

This paper uses a classical approach to feature selection: minimization of a cost function applied on estimated joint distributions. However, in this new formulation, the optimization search space is extended. The original search space is the Boolean lattice of features sets (BLFS), while the extended one is a collection of Boolean lattices of ordered pairs (CBLOP), that is (features, associated value), indexed by the elements of the BLFS. In this approach, we may not only select the features that are most related to a variable Y, but also select the values of the features that most influence the variable or that are most prone to have a specific value of Y. A local formulation of Shannon's mutual information, which generalizes Shannon's original definition, is applied on a CBLOP to generate a multiple resolution scale for characterizing variable dependence, the Local Lift Dependence Scale (LLDS). The main contribution of this paper is to define and apply the LLDS to analyse local properties of joint distributions that are neglected by the classical Shannon's global measure in order to select features. This approach is applied to select features based on the dependence between: i-the performance of students on university entrance exams and on courses of their first semester in the university; ii-the congress representative party and his vote on different matters; iii-the cover type of terrains and several terrain properties.

Local Lift Dependence Scale

“…which is defined for all (x, y) ∈ A c 1 and all (x, y) ∈ A 1 such that the second Radon-Nikodym derivative dµ/dν 1 (x, y) is well-defined, in which ν 1 is given by (10). Therefore, unless supp µ 2 is a fractal set, i.e., has a non-integer Hausdorff dimension, or A 1 is such that H 1 | A 1 is not σ-finite, the Lift Function is well-defined and can be calculated by means of line integrals.…”

“…In this framework, instead of expecting n × P(Y = 1) desired answers, we will expect n × P(Y = 1 | X = x opt ), which is [L(x opt , 1) − 1] × n more answers. This example may be extended to the continuous case, in which we want to maximize the answers in a subset of R and may choose profiles also in a subset of R. For an application of the Lift Function in statistics see [10].…”

“…We believe that this paper contributes to the state-of-art of variable dependence assessment, proposing a quite general local dependence quantifier which is applicable to a large class of joint probability distributions. We believe that there are much more facets to the Lift Function which should be explored, and that it could be of use not only to assess variable dependence, but for applications in areas such Stochastic Processes and Machine Learning (see [10] for example).…”

Overview of the 37th MaxEnt

“…Feature Selection Based on the Local Lift Dependence Scale [ 7 ] proposed a method for feature selection that is based on minimizing a cost function over an estimate of the observables’ joint distribution. While previous approaches perform a search over a Boolean lattice of feature sets (BLFS), the proposed method searched over a collection of Boolean lattices of ordered pairs (CBLOP).…”