An Integral Representation of the Logarithmic Function with Applications in Information Theory

Abstract

We explore a well-known integral representation of the logarithmic function, and demonstrate its usefulness in obtaining compact, easily-computable exact formulas for quantities that involve expectations and higher moments of the logarithm of a positive random variable (or the logarithm of a sum of i.i.d. positive random variables). The integral representation of the logarithm is proved useful in a variety of information-theoretic applications, including universal lossless data compression, entropy and differential entropy evaluations, and the calculation of the ergodic capacity of the single-input, multiple-output (SIMO) Gaussian channel with random parameters (known to both transmitter and receiver). This integral representation and its variants are anticipated to serve as a useful tool in additional applications, as a rigorous alternative to the popular (but non-rigorous) replica method (at least in some situations).

On Relations Between the Relative Entropy and χ2-Divergence, Generalizations and Applications

“…The entropy of a Poisson distribution, with parameter , is given by the integral representation [ 39 , 40 , 41 ] …”

Expected Logarithm and Negative Integer Moments of a Noncentral χ2-Distributed Random Variable

“…The recently proposed trick of representing the logarithm by an integral [ 7 ] turned out to be very helpful in the proof of the continuity of the expected logarithm (see Appendix A.3 ). While in general very well behaved, the logarithmic function nevertheless is a fickle beast due to its unboundedness both at zero and infinity.…”

“…Unfortunately, neither can be applied directly because is not nonnegative and unbounded both above and below. Instead we rely on a trick recently presented in [ 7 ] that allows us to write the expected logarithm with the help of the MGF: So, using the MGF of , we have We use this to prove continuity as follows. Assume that for some arbitrary large, but finite constant .…”

“…So, for example, in the field of information theory, there is a close relationship between the expected logarithm and entropy, and thus the expected logarithm of a noncentral -distributed RV plays an important role, e.g., in the description of the capacity of multiple-input, multiple-output noncoherent fading channels [ 1 , 2 ]. Many more examples in the field of information theory can be found in [ 7 ].…”

Some Useful Integral Representations for Information-Theoretic Analyses

“…In mathematical analyses associated with many problems in information theory and related fields, one is often faced with the need to compute expectations of logarithmic functions of composite random variables (see, e.g., [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ]), or moments of such random variables, whose order may be a general positive real, not even necessarily an integer (see, e.g., [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 ]).…”