From patterned response dependency to structured covariate dependency: Entropy based categorical-pattern-matching

Abstract

Data generated from a system of interest typically consists of measurements on many covariate features and possibly multiple response features across all subjects in a designated ensemble. Such data is naturally represented by one response-matrix against one covariate-matrix. A matrix lattice is an advantageous platform for simultaneously accommodating heterogeneous data types: continuous, discrete and categorical, and exploring hidden dependency among/between features and subjects. After each feature being individually renormalized with respect to its own histogram, the categorical version of mutual conditional entropy is evaluated for all pairs of response and covariate features according to the combinatorial information theory. Then, by applying Data Could Geometry (DCG) algorithmic computations on such a mutual conditional entropy matrix, multiple synergistic feature-groups are partitioned. Distinct synergistic feature-groups embrace distinct structures of dependency. The explicit details of dependency among members of synergistic features are seen through mutliscale compositions of blocks computed by a computing paradigm called Data Mechanics. We then propose a categorical pattern matching approach to establish a directed associative linkage: from the patterned response dependency to serial structured covariate dependency. The graphic display of such a directed associative linkage is termed an information flow and the degrees of association are evaluated via tree-to-tree mutual conditional entropy. This new universal way of discovering system knowledge is illustrated through five data sets. In each case, the emergent visible heterogeneity is an organization of discovered knowledge.

Unraveling the Regional Specificities of Malbec Wines from Mendoza, Argentina, and from Northern California

“…For a comprehensive description of this step, including a presentation of mutual entropy, we refer the reader to Fushing et al [26]. We note that entropy is a quantitative measure of "disorder", or randomness of a thermodynamic system.…”

“…The procedure is then repeated over all features and the values given for the gaps are then chosen so that the overall scale of the digital codes over the 69 characteristics of the wines is from 1 to 10. Once the digital code is established, a distance between two features j and k is computed by comparing the clustering of the wines that they produce, using mutual entropy as a distance measure (see Method Section and [26] for how the mutual entropy is computed). Using this distance, the 69 characteristics are then clustered using the DCG method [22,23].…”

“…We note that this method is related to the concept of feature selection [21], although it expands upon selection as it attempts to identify communities within features, rather than selecting one group of those features. The whole procedure is derived from previous work from the authors [22][23][24][25][26].…”

Unraveling Pattern-Based Mechanics Defining Self-Organized Recurrent Behaviors in a Complex System: A Zebrafish's Calcium Brain-Wide Imaging Example

“…To bring out these synergistic groups of pixels, we apply mutual conditional entropy from Combinatorial Information Theory (see [17]) to measure the degree of synergistic association between two pixels' calcium intensity time series. Basically in order to compute mutual conditional entropy, one possiblygapped histogram (see [14]) is derived for each of the two time series, excluding their ictal periods.…”

“…To address this issue, we calculate the mutual conditional-entropy among five clustering compositions, each of which respectively corresponds to one 7cluster tree-level of one computed HC-tree, pertaining to five inter-ictal periods. That is, each HC-tree renders one categorical variable with 7 categories in each inter-ictal period (see [17]).…”