Scalable Vector Gaussian Information Bottleneck

Mahvari, Mohammad Mahdi; Kobayashi, Mari; Zaidi, Abdellatif

doi:10.1109/isit45174.2021.9517720

Cited by 2 publications

(2 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More general frameworks could allow for variations in either the source of information (as in the case in temporal series) or the target variable (as is the case in transfer learning). Frameworks for both these kinds of extensions have already been proposed [ 46 , 78 ], and it would be interesting to study if, in these cases as well, the specific nature of the IB problem imprints the informationally optimal limits of several-stage processing.…”

Section: Limitations and Future Workmentioning

confidence: 99%

Exact and Soft Successive Refinement of the Information Bottleneck

Charvin,

Catenacci Volpi,

Polani

2023

Entropy

View full text Add to dashboard Cite

The information bottleneck (IB) framework formalises the essential requirement for efficient information processing systems to achieve an optimal balance between the complexity of their representation and the amount of information extracted about relevant features. However, since the representation complexity affordable by real-world systems may vary in time, the processing cost of updating the representations should also be taken into account. A crucial question is thus the extent to which adaptive systems can leverage the information content of already existing IB-optimal representations for producing new ones, which target the same relevant features but at a different granularity. We investigate the information-theoretic optimal limits of this process by studying and extending, within the IB framework, the notion of successive refinement, which describes the ideal situation where no information needs to be discarded for adapting an IB-optimal representation’s granularity. Thanks in particular to a new geometric characterisation, we analytically derive the successive refinability of some specific IB problems (for binary variables, for jointly Gaussian variables, and for the relevancy variable being a deterministic function of the source variable), and provide a linear-programming-based tool to numerically investigate, in the discrete case, the successive refinement of the IB. We then soften this notion into a quantification of the loss of information optimality induced by several-stage processing through an existing measure of unique information. Simple numerical experiments suggest that this quantity is typically low, though not entirely negligible. These results could have important implications for (i) the structure and efficiency of incremental learning in biological and artificial agents, (ii) the comparison of IB-optimal observation channels in statistical decision problems, and (iii) the IB theory of deep neural networks.

show abstract

Section: Limitations and Future Workmentioning

confidence: 99%

Exact and Soft Successive Refinement of the Information Bottleneck

Charvin,

Catenacci Volpi,

Polani

2023

Entropy

View full text Add to dashboard Cite

show abstract

“…In [19], the authors propose a variation of the original Information Bottleneck problem named Scalable Information Bottleneck, where multiple compressed representations, with increasingly richer features, are considered. The work in [20] proposes a solution for a distributed implementation of the IB problem, that is suitable in both the discrete and Gaussian case.…”

mentioning

confidence: 99%

Opportunistic Information-Bottleneck for Goal-Oriented Feature Extraction and Communication

Binucci,

Banelli,

Di Lorenzo

et al. 2024

IEEE Open J. Commun. Soc.

View full text Add to dashboard Cite

The Information Bottleneck (IB) method is an information theoretical framework to design a parsimonious and tunable feature-extraction mechanism, such that the extracted features are maximally relevant to a specific learning or inference task. Despite its theoretical value, the IB is based on a functional optimization problem that admits a closed form solution only on specific cases (e.g., Gaussian distributions), making it difficult to be applied in most applications, where it is necessary to resort to complex and approximated variational implementations. To overcome this limitation, we propose an approach to adapt the closedform solution of the Gaussian IB to a general task. Whichever is the inference task to be performed by a (possibly deep) neural-network, the key idea is to opportunistically design a regression sub-task, embedded in the original problem, where we can safely assume a (joint) multivariate normality between the sub-task's inputs and outputs. In this way we can exploit a fixed and pre-trained neural network to process the input data, using a tunable number of features, to trade data-size and complexity for accuracy. This approach is particularly useful every time a device needs to transmit data (or features) to a server that has to fulfil an inference task, as it provides a principled way to extract the most relevant features for the task to be executed, while looking for the best trade-off between the size of the feature vector to be transmitted, inference accuracy, and complexity. Extensive simulation results testify the effectiveness of the proposed method and encourage to further investigate this research line.

show abstract

Scalable Vector Gaussian Information Bottleneck

Cited by 2 publications

References 20 publications

Exact and Soft Successive Refinement of the Information Bottleneck

Exact and Soft Successive Refinement of the Information Bottleneck

Opportunistic Information-Bottleneck for Goal-Oriented Feature Extraction and Communication

Contact Info

Product

Resources

About