Near-Optimal Learning of Tree-Structured Distributions by Chow-Liu

Abstract

We provide finite sample guarantees for the classical Chow-Liu algorithm (IEEE Trans. Inform. Theory, 1968) to learn a tree-structured graphical model of a distribution. For a distribution P on Σ n and a tree T on n nodes, we say T is an ε-approximate tree for P if there is a T -structured distribution Q such that D(P Q) is at most ε more than the best possible tree-structured distribution for P . We show that if P itself is tree-structured, then the Chow-Liu algorithm with the plug-in estimator for mutual information with O(|Σ| 3 nε −1 ) i.i.d. samples outputs an ε-approximate tree for P with constant probability. In contrast, for a general P (which may not be tree-structured), Ω(n 2 ε −2 ) samples are necessary to find an ε-approximate tree. Our upper bound is based on a new conditional independence tester that addresses an open problem posed by Canonne, Diakonikolas, Kane, and Stewart (STOC, 2018): we prove that for three random variables X, Y, Z each over Σ, testing if I(X; Y | Z) is 0 or ≥ ε is possible with O(|Σ| 3 /ε) samples. Finally, we show that for a specific tree T , with O(|Σ| 2 nε −1 ) samples from a distribution P over Σ n , one can efficiently learn the closest T -structured distribution in KL divergence by applying the add-1 estimator at each node. * Höffgen's result is slightly different in that he doesn't use the plug-in estimator for mutual information.

Chow-Liu++: Optimal Prediction-Centric Learning of Tree Ising Models

“…Challenge 1: Failure of Chow-Liu. The main difficulty in proving our result is that the classical Chow-Liu algorithm, which is well-known to be optimal for structure learning and was recently shown to be optimal for total variation [6,13], does not work for learning to within locTV.…”

“…To prove this theorem we construct a distribution that is close to a tree Ising model in local total variation, such that even when given population (infinite sample) values for the correlations the Chow-Liu algorithm makes local errors that accumulate at global scales. This issue does not arise in the analysis of [6,13], since they assume a model is -close in total variation to a tree model, and thus the input distribution is extremely close to a tree model in a global sense. Furthermore, in converting this to a finite sample result they assume access to at least n log n samples, which results in extremely small local estimation errors that do not appreciably accumulate.…”

“…The statistical rates analyzed by [14] are achieved by exhaustive search algorithms and the task of finding efficient algorithms remained. Recently Bhattacharyya et al [6] and Daskalakis and Pan [13] studied exactly this problem, showing that the classical Chow-Liu algorithm achieves the optimal sample complexity O(n log n). They also showed robustness to model misspecification.…”

“…Instead of learning a model close in local total variation, one might use (global) total variation as done in the recent papers [6,13] which determined the optimal sample complexity for learning tree-structured models to within small total variation. Total variation distance also translates to a guarantee on the accuracy of posteriors P (X i |X S ), but does not exploit the small size of S. Indeed, as we show in Section D, learning to within small total variation has a high sample complexity of at least Ω(n log n) for trees on n nodes, and does not yield meaningful implications in the highdimensional setting of modern interest (where number of samples is much smaller than number of variables).…”

“…Total variation distance also translates to a guarantee on the accuracy of posteriors P (X i |X S ), but does not exploit the small size of S. Indeed, as we show in Section D, learning to within small total variation has a high sample complexity of at least Ω(n log n) for trees on n nodes, and does not yield meaningful implications in the highdimensional setting of modern interest (where number of samples is much smaller than number of variables). Furthermore, both [6] and [13] are based on the Chow-Liu algorithm, and as we will show in this paper, the Chow-Liu algorithm can be arbitrarily suboptimal with respect to local total variation. Thus, accuracy with respect to local versus global total variation results in a completely different learning problem with distinct algorithmic considerations.…”