Estimation of the information by an adaptive partitioning of the observation space

Abstract

Abstract-We demonstrate that it is possible to approximate the mutual information arbitrarily closely in probability by calculating relative frequencies on appropriate partitions and achieving conditional independence on the rectangles of which the partitions are made. Empirical results, including a comparison with maximum-likelihood estimators, are presented.

Estimating determinism rates to detect patterns in geospatial datasets

“…Our goal is to describe the stochastic and deterministic parts of the dynamics separately. For this, we develop a new nonparametric approach that combines EMD and continuous Mutual Information (Darbellay & Vajda, 1999) to assess all IMFs in order to automatically detect the cutoff point that allows us to combine them to form these two components. After obtaining the stochastic and deterministic components by using our approach, their influences can be estimated by means of a determinism rate, which is also proposed in this work.…”

“…We present an approach to decompose time series into stochastic and deterministic components based on Empirical Mode Decomposition (EMD), Fourier Transformation (FT), and the Mutual Information (MI) estimator proposed by Darbellay and Vajda (DV) (Darbellay & Vajda, 1999). In the context of this work, FT is not used to decompose time series, but to calculate the phase components of IMFs, instead.…”

Model structure selection using an integrated forward orthogonal search algorithm assisted by squared correlation and mutual information

“…If y is chosen to be the system output (the response), and x is one regressor in a linear model, can be used to measure the coherence of x with y in the model. Several algorithms have been proposed to estimate mutual information from observed data, see for example Moddemeijer (1989Moddemeijer ( , 1999, Darbellay andVajda (1999), andPaninski (2003) and the references therein. In this study, the adaptive partitioning histogram method proposed in Darbellay and Vajda (1999) was employed to estimate relevant mutual information.…”

A new maximum relevance-minimum multicollinearity (MRmMC) method for feature selection and ranking

“…Hence, an estimation of the mutual information is required and different methods can be employed. Among the possible methods are histogram-based [25], kernel density estimation [26], knearest neighbour [27], Parzen window [28], B-spline [29], adaptive partitioning [30,31] and fuzzy-based [32] approaches. These estimation methods typically involve some pre-set parameters whose optimal values heavily depend on problem characteristics.…”