Compact Modeling of Data Using Independent Variable Group Analysis

Abstract

In this paper, we introduce a modeling approach called independent variable group analysis (IVGA) which can be used for finding an efficient structural representation for a given data set. The basic idea is to determine such a grouping for the variables of the data set that mutually dependent variables are grouped together whereas mutually independent or weakly dependent variables end up in separate groups. Computation of an IVGA model requires a combinatorial algorithm for grouping of the variables and a modeling algorithm for the groups. In order to be able to compare different groupings, a cost function which reflects the quality of a grouping is also required. Such a cost function can be derived, for example, using the variational Bayesian approach, which is employed in our study. This approach is also shown to be approximately equivalent to minimizing the mutual information between the groups. The modeling task is computationally demanding. We describe an efficient heuristic grouping algorithm for the variables and derive a computationally light nonlinear mixture model for modeling of the dependencies within the groups. Finally, we carry out a set of experiments which indicate that IVGA may turn out to be beneficial in many different applications.Index Terms-Compact modeling, independent variable group analysis (IVGA), mutual information, variable grouping, variational Bayesian learning.

A comparison of algorithms for separation of synchronous subspaces

“…This is a valid approach which has seen some use; however, in general, this approach is a combinatorial optimization problem [10], since one does not know which of the estimated sources should be grouped together [11,12]. One must then either test all possible groupings, which grow very quickly with K and rapidly become intractable, or solve a discrete optimization problem by following, e.g., a greedy approach.…”

“…, K. This computation is non-trivial for N k ≥ 2 [13], further increasing the computational complexity of this approach. Nevertheless, this approach has been tackled, e.g., by estimating the entropy of multi-dimensional components using minimum spanning trees [12], or using variational Bayes approaches [11].…”

Agglomerative independent variable group analysis

“…IVGA is a method of grouping mutually dependent variables together while keeping mutually independent variables in separate groups [13,1]. This is done by maximising a bound on the marginal likelihood of a set of models enforcing the grouping.…”

“…This is in turn approximately equal to minimising an approximation of the mutual information between the groups. The original IVGA algorithm [13,1] uses a heuristic combinatorial hill climbing search to find the optimal grouping [13]. In this method, the size of the resulting grouping is not predetermined but optimised during the learning process.…”

“…Grouping or clustering variables based on their mutual dependence was the objective of the Independent Variable Group Analysis [13,1] (IVGA) method. In this paper we extend this approach by considering hierarchical grouping or clustering of variables.…”