Using the Semantic Information G Measure to Explain and Extend Rate-Distortion Functions and Maximum Entropy Distributions

Abstract

In the rate-distortion function and the Maximum Entropy (ME) method, Minimum Mutual Information (MMI) distributions and ME distributions are expressed by Bayes-like formulas, including Negative Exponential Functions (NEFs) and partition functions. Why do these non-probability functions exist in Bayes-like formulas? On the other hand, the rate-distortion function has three disadvantages: (1) the distortion function is subjectively defined; (2) the definition of the distortion function between instances and labels is often difficult; (3) it cannot be used for data compression according to the labels’ semantic meanings. The author has proposed using the semantic information G measure with both statistical probability and logical probability before. We can now explain NEFs as truth functions, partition functions as logical probabilities, Bayes-like formulas as semantic Bayes’ formulas, MMI as Semantic Mutual Information (SMI), and ME as extreme ME minus SMI. In overcoming the above disadvantages, this paper sets up the relationship between truth functions and distortion functions, obtains truth functions from samples by machine learning, and constructs constraint conditions with truth functions to extend rate-distortion functions. Two examples are used to help readers understand the MMI iteration and to support the theoretical results. Using truth functions and the semantic information G measure, we can combine machine learning and data compression, including semantic compression. We need further studies to explore general data compression and recovery, according to the semantic meaning.

Reviewing Evolution of Learning Functions and Semantic Information Measures for Understanding Deep Learning

“…where dij = d(xi, yj), s ≤ 0, and Zi is the partition function. In the author's opinion, exp(sdij) is a truth function, and this MinMI can be regarded as SeMI [18]. In 1967, Theil [44] put forward the generalized KL information formula:…”

“…In 2021, the author extended the information rate-distortion function and the maximum entropy method by replacing distortion function d(x, yj) with −logT(θj|x) and the average distortion with H(Yθ|X) [18]. In this way, the constraint is more flexible.…”

“…To extend the information rate-distortion function R(D), we replace the distortion function d(x, yj) with semantic information I(x; θj) and replace the upper limit D of average distortion d with the lower limit G of SeMI. Then R(D) becomes the information rate-fidelity function R(G) [13,18] (see Figure 4). Finally, following the deduction for R(D), we obtain the R(G) function with parameter s:…”

“…G/R indicates information efficiency. We can apply the R(G) function to image compression according to visual discrimination [16], semantic compression [18], and the convergence proofs for the MaxMI classification of unseen instances [13] and mixture models [73]. In addition, the author proves that the MaxMI test is equivalent to the maximum likelihood ratio test [76].…”

“…In R(G), R is the lower limit of ShMI for a given G, and G is the upper limit of SeMI for a given R. The R(G) function can reveal the relationship between SeMI and ShMI for understanding deep learning. Its applications to machine learning can be found in [12,13,18]. The main difference between semantic information and Shannon's information is that the former involves truth and falsehood, as discussed by Floridi [19].…”