Neural Estimation and Optimization of Directed Information over Continuous Spaces

Abstract

This work develops a new method for estimating and optimizing the directed information rate between two jointly stationary and ergodic stochastic processes. Building upon recent advances in machine learning, we propose a recurrent neural network (RNN)-based estimator which is optimized via gradient ascent over the RNN parameters. The estimator does not require prior knowledge of the underlying joint and marginal distributions. The estimator is also readily optimized over continuous input processes realized by a deep generative model. We prove consistency of the proposed estimation and optimization methods and combine them to obtain end-to-end performance guarantees. Applications for channel capacity estimation of continuous channels with memory are explored, and empirical results demonstrating the scalability and accuracy of our method are provided. When the channel is memoryless, we investigate the mapping learned by the optimized input generator.

Optimizing Estimated Directed Information over Discrete Alphabets

“…The DINE [22] is an RNN-based estimator of I(X → Y) from a sample D n := (X n , Y n ) ∼ P X n Y n . Its derivation begins with a representation of DI rate as the asymptotic difference of the following KL divergence terms:…”

“…, or in their parametrized form g θ , where θ ∈ Θ. With this notation, the DINE objective is given by [22]…”

“…The DINE architecture is portrayed in Figure 1. For formal consistency guarantees for DINE, as well as implementation details, the reader is referred to [22].…”

“…For memoryless channels, joint estimationoptimization methods over continuous input spaces were proposed in [20,21]. The case of channels with memory was recently treated in [22] using the DI neural estimator (DINE) developed therein. The DINE parametrizes the Donsker-Varadhan representation of DI by recurrent neural networks (RNNs), approximates expectations by sample means, and optimizes the resulting objective over the parameter space.…”

“…The DINE parametrizes the Donsker-Varadhan representation of DI by recurrent neural networks (RNNs), approximates expectations by sample means, and optimizes the resulting objective over the parameter space. To compute the feedback capacity, [22] further proposed an RNN-based generative model for continuous input distributions and jointly optimized it with DINE by propagating gradients through both models. These methods hinge on the end-to-end differentiability of the joint model, which fails to hold for discrete input alphabets.…”