The Bethe and Sinkhorn approximations of the pattern maximum likelihood estimate and their connections to the Valiant-Valiant estimate

Abstract

Abstract-For estimating a source's distribution histogram, Orlitsky and co-workers have proposed the pattern maximum likelihood (PML) estimate, which says that one should choose the distribution histogram that has the largest likelihood of producing the pattern of the observed symbol sequence. It can be shown that finding the PML estimate is equivalent to finding the distribution histogram that maximizes the permanent of a certain non-negative matrix.However, in general this optimization problem appears to be intractable and so one has to compute suitable approximations of the PML estimate. In this paper, we discuss various efficient PML estimate approximation algorithms, along with their connections to the Valiant-Valiant estimate of the distribution histogram. These connections are established by associating an approximately doubly stochastic matrix with the Valiant-Valiant estimate and comparing this approximately doubly stochastic matrix with the doubly stochastic matrices that appear in the free energy descriptions of the PML estimate and its approximations.

The Bethe and Sinkhorn Permanents of Low Rank Matrices and Implications for Profile Maximum Likelihood

“…samples from a discrete distribution, achieve an approximation factor of exp (−O( √ n log n)), improving upon the previous best-known bound achievable in polynomial time of exp(−O(n 2/3 log n)) (Charikar, Shiragur and Sidford, 2019). Through the work of Acharya, Das, Orlitsky and Suresh (2016), this implies a polynomial time universal estimator for symmetric properties of discrete distributions in a broader range of error parameter.We achieve these results by providing new bounds on the quality of approximation of the Bethe and Sinkhorn permanents (Vontobel, 2012 and2014). We show that each of these are exp(O(k log(N/k))) approximations to the permanent of N × N matrices with non-negative rank at most k, improving upon the previous known bounds of exp(O(N )).…”

“…We achieve these results by providing new bounds on the quality of approximation of the Bethe and Sinkhorn permanents (Vontobel, 2012 and2014). We show that each of these are exp(O(k log(N/k))) approximations to the permanent of N × N matrices with non-negative rank at most k, improving upon the previous known bounds of exp(O(N )).…”

“…We provide an efficient universal estimator that is based on the plug-in approach applied to the profile maximum likelihood (PML) distribution introduced by Orlitsky et al [OSS + 04]: given a sequence of n samples, PML is the distribution that maximizes the likelihood of the observed profile. The problem of computing the PML distribution has been studied in several papers since, applying heuristic approaches such as Bethe/Sinkhorn approximation [Von12,Von14], the EM algorithm [OSS + 04], a dynamic programming [PJW17] and algebraic methods [ADM + 10].…”

The Broad Optimality of Profile Maximum Likelihood

“…A number of different approaches have been taken to computing the PML and its approximations. Among the existing works, Acharya et al [1] considered exact algebraic computation, Orlitsky et al [61,62] designed an EM algorithm with MCMC acceleration, Vontobel [82,83] proposed a Bethe approximation heuristic, Anevski et al [7] introduced a sieved PML estimator and a stochastic approximation of the associated EM algorithm, and Pavlichin et al [68] derived a dynamic programming approach. Notably and recently, for a sample size n, Charikar et al [21] constructed an explicit exp(−O(n 2/3 log 3 n))-approximate PML whose computation time is near-linear in n.…”

Estimating a probability mass function with unknown labels

“…It is interesting to note that the likelihood (2) can be interpreted as a matrix permanent of the nonnegative matrix M ij := θ N j i . This relation enables one to use several techniques of approximate inference to evaluate the likelihood [25,26]. We will not pursue this idea further here.…”