Efficient Approximate Minimum Entropy Coupling of Multiple Probability Distributions

Li, Cheuk Ting

doi:10.1109/tit.2021.3076986

Cited by 8 publications

(15 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While much more challenging to optimize due to the concavity of entropy, there recently has been increasing interest in the minimization of entropy, particularly in the context of the minimum entropy coupling (MEC) problem. Recent applications of the MEC include the entropic causal inference framework [Kocaoglu et al, 2017a, Javidian et al, 2021; communications ; random number generation (discussed in [Li, 2021]); functional representation (discussed in [Cicalese et al, 2019]); and dimensionality reduction [Vidyasagar, 2012, Cicalese et al, 2016.…”

Section: Introductionmentioning

confidence: 99%

“…A variety of additional applications of minimum entropy couplings are discussed in [Cicalese et al, 2019, Li, 2021.…”

Section: Introductionmentioning

confidence: 99%

“…Many recent works have focused on proving approximation guarantees for the minimum entropy coupling problem. A unifying property of [Cicalese et al, 2019, Rossi, 2019, Li, 2021 is that they all show their approximation guarantee in relation to the same lower bound: the majorization lower bound. Moreover, shows that better guarantees for general m in relation to the majorization lower bound are impossible: thus establishing the majorization barrier.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Minimum-Entropy Coupling Approximation Guarantees Beyond the Majorization Barrier

Compton¹,

Katz²,

Qi³

et al. 2023

Preprint

View full text Add to dashboard Cite

Given a set of discrete probability distributions, the minimum entropy coupling is the minimum entropy joint distribution that has the input distributions as its marginals. This has immediate relevance to tasks such as entropic causal inference for causal graph discovery and bounding mutual information between variables that we observe separately. Since finding the minimum entropy coupling is NP-Hard, various works have studied approximation algorithms. The work of shows that the greedy coupling algorithm of [Kocaoglu et al., 2017a] is always within log 2 (e) ≈ 1.44 bits of the optimal coupling. Moreover, they show that it is impossible to obtain a better approximation guarantee using the majorization lower bound that all prior works have used: thus establishing a majorization barrier. In this work, we break the majorization barrier by designing a stronger lower bound that we call the profile method. Using this profile method, we are able to show that the greedy algorithm is always within log 2 (e)/e ≈ 0.53 bits of optimal for coupling two distributions (previous best-known bound is within 1 bit), and within 1+log 2 (e) 2 ≈ 1.22 bits for coupling any number of distributions (previous best-known bound is within 1.44 bits). We also examine a generalization of the minimum entropy coupling problem: Concave Minimum-Cost Couplings. We are able to obtain similar guarantees for this generalization in terms of the concave cost function. Additionally, we make progress on the open problem of [Kovačević et al., 2015] regarding NP membership of the minimum entropy coupling problem by showing that any hardness of minimum entropy coupling beyond NP comes from the difficulty of computing arithmetic in the complexity class NP. Finally, we present exponential-time algorithms for computing the exactly optimal solution. We experimentally observe that our new profile method lower bound is not only helpful for analyzing the greedy approximation algorithm, but also for improving the speed of our new backtracking-based exact algorithm.

show abstract

Section: Introductionmentioning

confidence: 99%

“…A variety of additional applications of minimum entropy couplings are discussed in [Cicalese et al, 2019, Li, 2021.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Minimum-Entropy Coupling Approximation Guarantees Beyond the Majorization Barrier

Compton¹,

Katz²,

Qi³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This has a variety of applications, including areas such as causal inference [1]- [4] and dimension reduction [5], [6]. In the context of random number generation as discussed in [7], the minimum entropy coupling is equivalent to determining the minimum entropy variable such that one sample from this variable enables us to generate one sample from any distribution of S.…”

Section: Introductionmentioning

confidence: 99%

“…[9] showed a 1-additive algorithm for m = 2 and ⌈log(m)⌉additive for general m. [1] introduced the greedy coupling algorithm, [2] showed this is a local optima and [10] showed this is a 1-additive algorithm for m = 2. Most recently, [7] introduced a new (2 − 2 2−m )-additive algorithm.…”

Section: Introductionmentioning

confidence: 99%

A Tighter Approximation Guarantee for Greedy Minimum Entropy Coupling

Compton¹

2022

Preprint

View full text Add to dashboard Cite

We examine the minimum entropy coupling problem, where one must find the minimum entropy variable that has a given set of distributions S = {p1, . . . , pm} as its marginals. Although this problem is NP-Hard, previous works have proposed algorithms with varying approximation guarantees. In this paper, we show that the greedy coupling algorithm of [Kocaoglu et al., AAAI'17] is always within log 2 (e) (≈ 1.44) bits of the minimum entropy coupling. In doing so, we show that the entropy of the greedy coupling is upper-bounded by H( S) + log 2 (e). This improves the previously best known approximation guarantee of 2 bits within the optimal [Li, IEEE Trans. Inf. Theory '21]. Moreover, we show our analysis is tight by proving there is no algorithm whose entropy is upper-bounded by H( S) + c for any constant c < log 2 (e). Additionally, we examine a special class of instances where the greedy coupling algorithm is exactly optimal.

show abstract

A Decomposition-Based Improved Broad Learning System Model for Short-Term Load Forecasting

Cheng

et al. 2022

J. Electr. Eng. Technol.

View full text Add to dashboard Cite

Efficient Approximate Minimum Entropy Coupling of Multiple Probability Distributions

Cited by 8 publications

References 38 publications

Minimum-Entropy Coupling Approximation Guarantees Beyond the Majorization Barrier

Minimum-Entropy Coupling Approximation Guarantees Beyond the Majorization Barrier

A Tighter Approximation Guarantee for Greedy Minimum Entropy Coupling

A Decomposition-Based Improved Broad Learning System Model for Short-Term Load Forecasting

Contact Info

Product

Resources

About