On the equivalence of double maxima and KL-means for information bottleneck-based source coding

Hassanpour, Shayan; Wübben, Dirk; Dekorsy, Armin

doi:10.1109/wcnc.2018.8377199

Cited by 9 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Details on its convergence are discussed in [ 3 ]. In addition to the iterative information bottleneck algorithm from [ 2 ], many other information bottleneck algorithms appeared in the literature (e.g., [ 57 , 58 ]). These algorithms can be understood as the work horses of the information bottleneck method, as they can determine the compression mapping

for a given

and a desired cardinality

.…”

Section: The Information Bottleneck Methods and Coarsely Quantized In...mentioning

confidence: 99%

“…In the very important special case of aiming to preserve a maximum desired amount of I

(i.e.,

for a given cardinality

), the clustering of

described by

becomes a hard clustering [ 58 ]. In this case, it is easy to limit the compression information I

by a proper choice of the cardinality

of the compressed representation.…”

Section: The Information Bottleneck Methods and Coarsely Quantized In...mentioning

confidence: 99%

See 1 more Smart Citation

Information Bottleneck Signal Processing and Learning to Maximize Relevant Information for Communication Receivers

Lewandowsky

Bauch

Stark

2022

Entropy

View full text Add to dashboard Cite

Digital communication receivers extract information about the transmitted data from the received signal in subsequent processing steps, such as synchronization, demodulation and channel decoding. Technically, the receiver-side signal processing for conducting these tasks is complex and hence causes bottleneck situations in terms of power, delay and chip area. Typically, many bits per sample are required to represent and process the received signal in the digital receiver hardware accurately. In addition, demanding arithmetical operations are required in the signal processing algorithms. A popular recent trend is designing entire receiver chains or some of their crucial building blocks from an information theoretical perspective. Signal processing blocks with very simple mathematical operations can be designed to directly maximize the relevant information that flows through them. At the same time, a strong quantization reduces the number of bits processed in the receiver to further lower the complexity. The described system design approach follows the principle of the information bottleneck method. Different authors proposed various ideas to design and implement mutual information-maximizing signal processing units. The first important aim of this article is to explain the fundamental similarities between the information bottleneck method and the functionalities of communication receivers. Based on that, we present and investigate new results on an entire receiver chain that is designed following the information bottleneck design principle. Afterwards, we give an overview of different techniques following the information bottleneck design paradigm from the literature, mainly dealing with channel decoding applications. We analyze the similarities of the different approaches for information bottleneck signal processing. This comparison leads to a general view on information bottleneck signal processing which goes back to the learning of parameters of trainable functions that maximize the relevant mutual information under compression.

show abstract

for a given

and a desired cardinality

.…”

Section: The Information Bottleneck Methods and Coarsely Quantized In...mentioning

confidence: 99%

“…In the very important special case of aiming to preserve a maximum desired amount of I

(i.e.,

for a given cardinality

), the clustering of

described by

becomes a hard clustering [ 58 ]. In this case, it is easy to limit the compression information I

by a proper choice of the cardinality

of the compressed representation.…”

Section: The Information Bottleneck Methods and Coarsely Quantized In...mentioning

confidence: 99%

Information Bottleneck Signal Processing and Learning to Maximize Relevant Information for Communication Receivers

Lewandowsky

Bauch

Stark

2022

Entropy

View full text Add to dashboard Cite

show abstract

“…The Information Bottleneck (Tishby, Pereira, and Bialek 2000;Dimitrov and Miller 2001;Samengo 2002) (IB) method provides a principled approach to this problem, which compresses the source random variable to keep the information relevant for predicting the target random variable while discarding all irrelevant information. The IB method has been applied to various domains such as classification (Hecht, Noor, and Tishby 2009), clustering (Slonim et al 2005), coding theory (Hassanpour, Wübben, and Dekorsy 2018;Zeitler et al 2008), and quantization (Strouse and Schwab 2017;Cheng et al 2019). Recent research also demonstrate that the IB method can produce well-generalized representations (Shamir, Sabato, and Tishby 2010;Vera, Piantanida, and Vega 2018;Amjad and Geiger 2019) and may be promising on explaining the learning behaviors of neural networks (Tishby and Zaslavsky 2015;Shwartz-Ziv and Tishby 2017;Saxe et al 2019).…”

Section: Introductionmentioning

confidence: 99%

Disentangled Information Bottleneck

Pan

Niu

Zhang

et al. 2021

AAAI

View full text Add to dashboard Cite

The information bottleneck (IB) method is a technique for extracting information that is relevant for predicting the target random variable from the source random variable, which is typically implemented by optimizing the IB Lagrangian that balances the compression and prediction terms. However, the IB Lagrangian is hard to optimize, and multiple trials for tuning values of Lagrangian multiplier are required. Moreover, we show that the prediction performance strictly decreases as the compression gets stronger during optimizing the IB Lagrangian. In this paper, we implement the IB method from the perspective of supervised disentangling. Specifically, we introduce Disentangled Information Bottleneck (DisenIB) that is consistent on compressing source maximally without target prediction performance loss (maximum compression). Theoretical and experimental results demonstrate that our method is consistent on maximum compression, and performs well in terms of generalization, robustness to adversarial attack, out-of-distribution detection, and supervised disentangling.

show abstract

“…We seek a coding scheme that generates a code

from the output of the channel X which is as informative as possible about the original source signal Y and can be transmitted at a small rate

. Therefore, this problem is equivalent to the the formulation of the information bottleneck [ 11 ].…”

Section: Introductionmentioning

confidence: 99%

The Convex Information Bottleneck Lagrangian

Rodríguez-Gálvez

Thobaben

Skoglund

2020

Entropy

View full text Add to dashboard Cite

The information bottleneck (IB) problem tackles the issue of obtaining relevant compressed representations T of some random variable X for the task of predicting Y . It is defined as a constrained optimization problem which maximizes the information the representation has about the task, I(T ; Y ), while ensuring that a certain level of compression r is achieved (i.e., I(X; T ) ≤ r). For practical reasons, the problem is usually solved by maximizing the IB Lagrangian (i.e., L IB (T ; β) = I(T ; Y ) − βI(X; T )) for many values of β ∈ [0, 1]. Then, the curve of maximal I(T ; Y ) for a given I(X; T ) is drawn and a representation with the desired predictability and compression is selected. It is known when Y is a deterministic function of X, the IB curve cannot be explored and another Lagrangian has been proposed to tackle this problem: the squared IB Lagrangian: L sq-IB (T ; β sq ) = I(T ; Y ) − β sq I(X; T ) 2 . In this paper, we (i) present a general family of Lagrangians which allow for the exploration of the IB curve in all scenarios; (ii) provide the exact one-to-one mapping between the Lagrange multiplier and the desired compression rate r for known IB curve shapes; and (iii) show we can approximately obtain a specific compression level with the convex IB Lagrangian for both known and unknown IB curve shapes. This eliminates the burden of solving the optimization problem for many values of the Lagrange multiplier. That is, we prove that we can solve the original constrained problem with a single optimization. F IB,max (r) = max T ∈∆ {I(T ; Y )} s.t. I(X; T ) ≤ r, ∀r ∈ [0, ∞).(1)Definition 2 (IB curve). The IB curve is the set of points defined by the solutions of F IB,max (r) for varying values of r ∈ [0, ∞).Definition 3 (Information plane). The plane is defined by the axes I(T ; Y ) and I(X; T ).This method has been successfully applied to solve different problems from a variety of domains. For example:• Supervised learning. In supervised learning, we are presented with a set of n pairs of input features and task outputs instances. We seek an approximation of the conditional probability distribution between the task

show abstract

On the equivalence of double maxima and KL-means for information bottleneck-based source coding

Cited by 9 publications

References 16 publications

Information Bottleneck Signal Processing and Learning to Maximize Relevant Information for Communication Receivers

Information Bottleneck Signal Processing and Learning to Maximize Relevant Information for Communication Receivers

Disentangled Information Bottleneck

The Convex Information Bottleneck Lagrangian

Contact Info

Product

Resources

About