A measure of statistical complexity based on predictive information with application to finite spin systems

Abdallah, Samer A.; Plumbley, Mark D.

doi:10.1016/j.physleta.2011.10.066

Cited by 40 publications

(53 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar duality was noted by [5] in relation to the multi-information and the binding information (the extensive counterpart to the predictive information rate) in finite sets of discrete-valued random variables.…”

Section: Process Information Measures For Gaussian Processessupporting

confidence: 64%

See 1 more Smart Citation

Predictive Information in Gaussian Processes with Application to Music Analysis

Abdallah

Plumbley

2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. We describe an information-theoretic approach to the analysis of sequential data, which emphasises the predictive aspects of perception, and the dynamic process of forming and modifying expectations about an unfolding stream of data, characterising these using a set of process information measures. After reviewing the theoretical foundations and the definition of the predictive information rate, we describe how this can be computed for Gaussian processes, including how the approach can be adpated to non-stationary processes, using an online Bayesian spectral estimation method to compute the Bayesian surprise. We finish with a sample analysis of a recording of Steve Reich's Drumming.

show abstract

Section: Process Information Measures For Gaussian Processessupporting

confidence: 64%

“…In previous work [5] we examined several process information measures and their interrelationships, as well as generalisation of these for arbitrary countable sets of random variables. Following the conventions established there, we let ← X t = (.…”

Section: Introductionmentioning

confidence: 99%

Predictive Information in Gaussian Processes with Application to Music Analysis

Abdallah

Plumbley

2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…dit implements the vast majority of information measure defined in the literature, including entropies (Shannon (Cover and Thomas 2006), Renyi, Tsallis), multivariate mutual informations (co-information (Bell 2003) (McGill 1954), total correlation (Watanabe 1960), dual total correlation(Te Sun 1980) (Han 1975) (Abdallah and Plumbley 2012), CAEKL mutual information (Chan et al 2015)), common informations (Gács-Körner(Gács and Körner 1973) (Tyagi, Narayan, and Gupta 2011), Wyner (Wyner 1975)(W. Liu, Xu, and Chen 2010), exact (Kumar, Li, and El Gamal 2014), functional, minimal sufficient statistic), and channel capacity (Cover and Thomas 2006). It includes methods of studying joint distributions including information diagrams, connected informations (Schneidman et al 2003) (Amari 2001), marginal utility of information (Allen, Stacey, and Bar-Yam 2017), and the complexity profile(Y.…”

mentioning

confidence: 99%

dit: a Python package for discrete information theory

James¹,

Ellison²,

Crutchfield³

2018

JOSS

View full text Add to dashboard Cite

Available at: Https://Github.com/Dit/Dit" n.d.) is a Python package for the study of discrete information theory. Information theory is a mathematical framework for the study of quantifying, compressing, and communicating random variables (Cover and Thomas 2006) (MacKay 2003) (Yeung 2008). More recently, information theory has been utilized within the physical and social sciences to quantify how different components of a system interact. dit is primarily concerned with this aspect of the theory.Information theory is a powerful extension to probability and statistics, quantifying dependencies among arbitrary random variables in a way tha tis consistent and comparable across systems and scales. Information theory was originally developed to quantify how quickly and reliably information could be transmitted across an arbitrary channel. The demands of modern, data-driven science have been coopting and extending these quantities and methods into unknown, multivariate settings where the interpretation and best practices are not known. For example, there are at least four reasonable multivariate generalizations of the mutual information, none of which inherit all the interpretations of the standard bivariate case. Which is best to use is context-dependent. dit implements a vast range of multivariate information measures in an effort to allow information practitioners to study how these various measures behave and interact in a variety of contexts. We hope that having all these measures and techniques implemented in one place will allow the development of robust techniques for the automated quantification of dependencies within a system and concrete interpretation of what those dependencies mean.dit implements the vast majority of information measure defined in the literature, including entropies (Shannon(Cover and

show abstract

“…• Similarly, when marginal independence holds, we see that I P X|Y = 0 from (20). Otherwise stated, H P X|Y = H P X and H P Y|X = H P Y .…”

Section: The Aggregate and Split Channel Multivariate Balance Equationmentioning

confidence: 86%

“…An important point to realize is that the multivariate transmitted information between two different random vectors I P XY is the proper generalization for the usual mutual information MI P XY in the bivariate case, rather than the more complex alternatives used in multivariate sources (see Section 2.2 and [5,14]). Indeed properties (18) and (20) are crucial in transporting the structure and intuitions built from the bivariate channel entropy triangle to the multivariate one, of which the former is a proper instance. This was not the case with balance equations and entropy triangles for stochastic sources of information [5].…”

Section: Discussionmentioning

confidence: 99%

Assessing Information Transmission in Data Transformations with the Channel Multivariate Entropy Triangle

Valverde-Albacete

Peláez-Moreno

2018

Entropy

View full text Add to dashboard Cite

Data transformation, e.g., feature transformation and selection, is an integral part of any machine learning procedure. In this paper, we introduce an information-theoretic model and tools to assess the quality of data transformations in machine learning tasks. In an unsupervised fashion, we analyze the transformation of a discrete, multivariate source of information X into a discrete, multivariate sink of information Y related by a distribution P XY . The first contribution is a decomposition of the maximal potential entropy of (X, Y), which we call a balance equation, into its (a) non-transferable, (b) transferable, but not transferred, and (c) transferred parts. Such balance equations can be represented in (de Finetti) entropy diagrams, our second set of contributions. The most important of these, the aggregate channel multivariate entropy triangle, is a visual exploratory tool to assess the effectiveness of multivariate data transformations in transferring information from input to output variables. We also show how these decomposition and balance equations also apply to the entropies of X and Y, respectively, and generate entropy triangles for them. As an example, we present the application of these tools to the assessment of information transfer efficiency for Principal Component Analysis and Independent Component Analysis as unsupervised feature transformation and selection procedures in supervised classification tasks.

show abstract

A measure of statistical complexity based on predictive information with application to finite spin systems

Cited by 40 publications

References 23 publications

Predictive Information in Gaussian Processes with Application to Music Analysis

Predictive Information in Gaussian Processes with Application to Music Analysis

dit: a Python package for discrete information theory

Assessing Information Transmission in Data Transformations with the Channel Multivariate Entropy Triangle

Contact Info

Product

Resources

About