The Double-Sided Information-Bottleneck Function

Abstract

We consider a two-terminal variant (double-sided) of the information bottleneck problem, which is related to biclustering. In our setup, X and Y are dependent random variables and the problem is to find two independent channels P U|X and P V|Y (setting the Markovian structure U → X → Y → V) that maximize I(U; V) subject to constraints on the relevant mutual information expressions: I(U; X) and I(V; Y). For jointly Gaussian X and Y, we show that Gaussian channels are optimal in the low-SNR regime, but not for general SNR. Similarly, it is shown that for a doubly symmetric binary source, binary symmetric channels are optimal when the correlation is low, and are suboptimal for high correlation. We conjecture that Z and S channels are optimal when the correlation is 1 (i.e., X = Y), and provide supporting numerical evidence.

Exact and Soft Successive Refinement of the Information Bottleneck

“…The proof of Proposition 5 exemplifies how our convex hull characterisation interlocks well with the convexity approach to the IB [35][36][37][38][39]. In this sense, our characterisation brings a new theoretical tool to the study of the successive refinement of the IB.…”

“…This characterisation is proven in the discrete case under an additional but mild assumption of injectivity of the decoder q(X|T). This new point of view fits well with an ongoing convexity approach to the IB problem [35][36][37][38][39] and might thus help develop a new geometric perspective on the successive refinement of the IB. As an example, we use this geometric characterisation to prove that SR always holds for binary source X and binary relevancy Y.…”

“…Then, we propose a new geometric characterisation of SR, which builds on the intuition that what is "known" by a bottleneck is the convex hull of its decoder conditional probabilities. This new point of view, associated with an active approach that reformulates the IB problem as that of finding the lower convex envelope of a well-chosen function [35][36][37][38][39], provides a new tool for theoretical research on this topic. We exemplify this potential fertility by proving, thanks to the combination of our convex hull characterisation with the convexity approach to the IB, the successive refinability of binary source X and binary relevancy Y (Proposition 5).…”

“…The geometric approach in which we propose to study the successive refinement of the IB is closely related to the convexity approach to the IB [35][36][37][38][39], which frames the IB problem as that of finding the lower convex hull of a well-chosen function. This formulation happens to fit neatly with our convex hull characterisation of successive refinement; we use it to apply the characterisation to proving successive refinability in the case of a binary source and relevancy.…”

“…Our convex hull characterisation intertwines the question of exact SR with the more fundamental question of the structure of decoder curves λ → q λ (X|t) , t ∈ T (11) on the source simplex ∆ X , a question for which the convexity approach to the IB problem [35][36][37][38][39] seems promising. In short, this approach reformulates the IB problem to that of finding the lower convex envelope of a well-chosen function F β , defined on the source simplex ∆ X , and parameterised by the information curve's inverse slope β (see Appendix B.7).…”