Efficient Estimation of Mutual Information for Strongly Dependent Variables

Gao, Shuyang; Steeg, Greg Ver; Galstyan, Aram

doi:10.48550/arxiv.1411.2003

Cited by 32 publications

(18 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figures 2 and 3, for the two and three component models, respectively, show the results of calculations for k = 5. For a detailed study of dependence on k we refer the readers to [27].…”

Section: Toy Model For Multiplicative Processesmentioning

confidence: 99%

Using Mutual Information to measure Time-lags from non-linear processes in Astronomy

Chakraborty¹,

Leeuwen²

2021

Preprint

View full text Add to dashboard Cite

Measuring time lags between time-series or lighcurves at different wavelengths from a variable or transient source in astronomy is an essential probe of physical mechanisms causing multiwavelength variability. Time-lags are typically quantified using discrete correlation functions (DCF) which are appropriate for linear relationships. However, in variable sources like X-ray binaries, active galactic nuclei (AGN) and other accreting systems, the radiative processes and the resulting multiwavelength lightcurves often have non-linear relationships. For such systems it is more appropriate to use non-linear information-theoretic measures of causation like mutual information, routinely used in other disciplines. We demonstrate with toy models the loopholes of using the standard DCF and show improvements when using the mutual information correlation function (MICF). For non-linear correlations, the latter accurately and sharply identifies the lag components as opposed to the DCF which can be erroneous. Following that we apply the MICF to the multi-wavelength lightcurves of AGN NGC 4593. We find that X-ray fluxes are leading UVW2 fluxes by ∼ 0.2 days, closer to model predictions from reprocessing by the accretion disk than the DCF estimate. The uncertainties with the current lightcurves are too large though to rule out -ve lags. Additionally, we find another delay component at ∼ −1 day i.e. UVW2 leading X-rays consistent with inward propagating fluctuations in the accretion disk scenario. This is not detected by the DCF. Keeping in mind the non-linear relation between X-ray and UVW2, this is worthy of further theoretical investigation. From both the toy models and real observations, it is clear that the mutual information based estimator is highly sensitive to complex non-linear correlations. With sufficiently high temporal resolution, we will precisely detect each of the lag features corresponding to these correlations.

show abstract

“…Figures 2 and 3, for the two and three component models, respectively, show the results of calculations for k = 5. For a detailed study of dependence on k we refer the readers to [27].…”

Section: Toy Model For Multiplicative Processesmentioning

confidence: 99%

Using Mutual Information to measure Time-lags from non-linear processes in Astronomy

Chakraborty¹,

Leeuwen²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The estimation of the Shannon mutual information from samples remains an active research problem. Lately, in theoretical as well as practical fronts, there has a resurgence of interest in entropy and mutual information estimators (see Sricharan et al (2013), Jiao et al (2014), Singh et Pøczos (2017), Singh and Póczos (2016), Moon et al (2017), Han et al (2015), Gao et al (2014), Gao et al (2015), Gao et al (2016a), Gao et al (2016b), Angeliki & Dimitris (2009, Walters et al (2009), and some offer good results even for small samples (Khan et al (2007)).…”

Section: E(pmentioning

confidence: 99%

Non parametric estimation of joint, Renyi-Stallis entropies and mutual information and asymptotic limits

Ba¹,

Lo²,

Seck³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The current solution is to use the Blahut-Arimoto algorithm [29], which essentially enumerates over all states, thus being limited to small-scale problems and not being applicable to the continuous domain. More scalable non-parametric estimators have been developed [7,6]: these have a high memory footprint or require a very large number of observations, any approximation may not be a bound on the MI making reasoning about correctness harder, and they cannot easily be composed with existing (gradient-based) systems that allow us to design a unified (end-to-end) system. In the continuous domain, Monte Carlo integration has been proposed [10], but applications of Monte Carlo estimators can require a large number of draws to obtain accurate solutions and manageable variance.…”

Section: Scalable Information Maximisationmentioning

confidence: 99%

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

Mohamed,

Rezende

2015

Preprint

View full text Add to dashboard Cite

The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents. Most learning algorithms that involve optimisation of the mutual information rely on the Blahut-Arimoto algorithm -an enumerative algorithm with exponential complexity that is not suitable for modern machine learning applications. This paper provides a new approach for scalable optimisation of the mutual information by merging techniques from variational inference and deep learning. We develop our approach by focusing on the problem of intrinsically-motivated learning, where the mutual information forms the definition of a well-known internal drive known as empowerment. Using a variational lower bound on the mutual information, combined with convolutional networks for handling visual input streams, we develop a stochastic optimisation algorithm that allows for scalable information maximisation and empowerment-based reasoning directly from pixels to actions.

show abstract

Efficient Estimation of Mutual Information for Strongly Dependent Variables

Cited by 32 publications

References 21 publications

Using Mutual Information to measure Time-lags from non-linear processes in Astronomy

Using Mutual Information to measure Time-lags from non-linear processes in Astronomy

Non parametric estimation of joint, Renyi-Stallis entropies and mutual information and asymptotic limits

Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

Contact Info

Product

Resources

About