Bottleneck Problems: An Information and Estimation-Theoretic View

Abstract

Information bottleneck (IB) and privacy funnel (PF) are two closely related optimization problems which have found applications in machine learning, design of privacy algorithms, capacity problems (e.g., Mrs. Gerber’s Lemma), and strong data processing inequalities, among others. In this work, we first investigate the functional properties of IB and PF through a unified theoretical framework. We then connect them to three information-theoretic coding problems, namely hypothesis testing against independence, noisy source coding, and dependence dilution. Leveraging these connections, we prove a new cardinality bound on the auxiliary variable in IB, making its computation more tractable for discrete random variables. In the second part, we introduce a general family of optimization problems, termed “bottleneck problems”, by replacing mutual information in IB and PF with other notions of mutual information, namely f-information and Arimoto’s mutual information. We then argue that, unlike IB and PF, these problems lead to easily interpretable guarantees in a variety of inference tasks with statistical constraints on accuracy and privacy. While the underlying optimization problems are non-convex, we develop a technique to evaluate bottleneck problems in closed form by equivalently expressing them in terms of lower convex or upper concave envelope of certain functions. By applying this technique to a binary case, we derive closed form expressions for several bottleneck problems.

A Class of Nonbinary Symmetric Information Bottleneck Problems

“…The following tight cardinality bound was established in [25]. It was actually already proved for the corresponding dual problem, namely the IB Lagrangian, in [26].…”

“…It also interesting to consider rather more classical use-cases, i.e, multi-user channel capacity and Noisy Source Coding problems. A comprehensive summary of the different relations between the IB and Privacy Funnel problems has been presented in [25].…”

The Double-Sided Information-Bottleneck Function

“…The following tight cardinality bound in the single-sided counterpart of our problem was established in [33].…”

The Distributed Information Bottleneck reveals the explanatory structure of complex systems

“…The Information Bottleneck (IB) is a promising framework to lend interpretability to deep learning and allow in-depth analysis of a relationship [1,20]. Given random variables X and Y serving as an input and an output, the IB defines a spectrum of compressed representations of X that retain only the most relevant information about Y .…”