On Approximating Total Variation Distance

Abstract

Total variation distance (TV distance) is a fundamental notion of distance between probability distributions. In this work, we introduce and study the problem of computing the TV distance of two product distributions over the domain {0,1}^n. In particular, we establish the following results. 1. The problem of exactly computing the TV distance of two product distributions is #P-complete. This is in stark contrast with other distance measures such as KL, Chi-square, and Hellinger which tensorize over the marginals leading to efficient algorithms. 2. There is a fully polynomial-time deterministic approximation scheme (FPTAS) for computing the TV distance of two product distributions P and Q where Q is the uniform distribution. This result is extended to the case where Q has a constant number of distinct marginals. In contrast, we show that when P and Q are Bayes net distributions the relative approximation of their TV distance is NP-hard.

A simple polynomial-time approximation algorithm for the total variation distance between two product distributions

“…Unlike many other quantities for similar uses, such as the relative entropy and the 2 -divergence, the TV distance does not tensorise over product distributions. In fact, it was discovered recently that, somewhat surprisingly, exact computation of the total variation distance, even between product distributions over the Boolean domain, is #P-hard [1].…”

“…is leaves open the question of approximation complexity of the TV distance. In [1], the authors give polynomial-time randomised approximation algorithms in two special cases over the Boolean domain, when one of the distribution has marginals over 1/2 and dominates the other, or when one of the distribution has a constant number of distinct marginals. eir method is based on Dyer's dynamic programming algorithm for approximating the number of knapsack solutions [2].…”

“…One remaining question is if a deterministic approximation algorithm exists for the TV distance. e answer might be positive, because of the connection with counting knapsack solutions established by Bha acharyya, Gayen, Meel, Myrisiotis, Pavan, and Vinodchandran [1], and the deterministic approximation algorithm for the la er problem by Štefankovič, Vempala, and Vigoda [4] and by Gopalan, Klivans, and Meka [3], independently.…”

“…e above equation holds because (1) for any valid coupling C, it holds that Pr C [ = = ] ≤ min{ ( ), ( )}; (2) to achieve the optimal coupling, every must achieve the equality. We have…”