On the Relevance-Complexity Region of Scalable Information Bottleneck

Exact and Soft Successive Refinement of the Information Bottleneck

“…Proof. This single-letter characterisation is a consequence of Remark 1 in [32], which states the following: a tuple (R 1 , . .…”

“…We propose a purely single-letter definition (i.e., we only consider single source, relevancy, and bottleneck variables), which makes the presentation simpler but still conveys the intuition of information incorporation. After having presented the notion of SR in the IB framework, we describe its Markov chain characterisation (see Proposition 1), which mirrors the characterisation of SR for classic rate-distortion problems [ 26 ], and makes our formulation equivalent to previous multi-letter operational definitions, which also formalise the intuition of information incorporation [ 30 , 31 , 32 ]. We then leverage this characterisation to prove SR in the case of Gaussian vectors and deterministic channel .…”

“…It turns out that our single-letter definition and the operational multi-letter definition from [ 30 , 31 ] are equivalent. The two latter works—as well as [ 32 ]—thus provide our single-letter definition with an operational interpretation that also formalises the intuition of an informationally optimal incorporation of information (see Proposition 1 and Appendix B.2 ).…”

“…More precisely, in the context outlined above, there is successive refinement if the processing cost of first producing a coarse bottleneck of parameter and then refining it to a finer bottleneck of parameter is no larger than the processing cost of directly producing a bottleneck of parameter without any intermediary bottleneck (see Section 2.1 and Appendix B.2 for formal definitions). The aim of this work is to push the understanding of successive refinement in the IB framework [ 30 , 31 , 32 ] further, as well as to expand the analysis to a quantification of the lack of SR, in cases where the latter does not hold exactly. We start by leveraging general results in existing IB literature [ 33 , 34 ] to prove that successive refinement always holds for jointly Gaussian , and when Y is a deterministic function of X .…”

“…Then, Ref. [ 32 ] goes further by considering the informationally optimal limits of several-stage processing in general, without comparing it to single-stage processing. In both these works, the problem is initially defined in asymptotic coding terms, and only then given a single-letter characterisation.…”