A Categorical Characterization of Relative Entropy on Standard Borel Spaces

Abstract

We give a categorical treatment, in the spirit of Baez and Fritz, of relative entropy for probability distributions defined on standard Borel spaces. We define a category called SbStat suitable for reasoning about statistical inference on standard Borel spaces. We define relative entropy as a functor into Lawvere's category [0, ∞] and we show convexity, lower semicontinuity and uniqueness.Recently there have been some exciting developments that bring some categorical insights to probability theory and specifically to learning theory. These are reported in some recent papers by Clerc, Dahlqvist, Danos and Garnier [DG15, DDG16, CDDG17]. The first of these papers showed how to view the Dirichlet distribution as a natural transformation thus opening the way to an understanding of higher-order probabilities, while the second gave a powerful framework for constructing several natural transformations. In [DG15] the hope was expressed that one could use these ideas to understand Bayesian inversion, a core concept in machine learning. In [CDDG17] this was realized in a remarkably novel way. These papers carry out their investigations in the setting of standard Borel spaces and are based on the Giry monad [Gir81, Law64].

A functorial characterization of von Neumann entropy

“…The relative entropy naturally appears as a byproduct of Lindblad's Lemma and our functorial characterization of the quantum entropy. However, this is not quite a functorial characterization of the quantum relative entropy, at least not in the spirit of the recent functorial characterizations of the relative Shannon entropy (or Kullback-Leibler divergence) existing in the literature [3,16,25]. Therefore, a natural question to ask is if the quantum relative entropy has a functorial characterization, generalizing the characterization of Baez and Fritz [3].…”

“…a There is, however, a natural stochastic section, but adding such morphisms would dramatically change the category. Such a stochastic section is an example of a disintegration [42], which is a crucial ingredient in the categorical classification of classical relative entropy [3,16]. We will also discuss disintegrations in the next section, as they are used in our main characterization theorem.…”

Towards a functorial description of quantum relative entropy

“…For example, optimal hypotheses are Bayesian inverses [8,Theorem 8.3], which admit stronger compositional properties [8,Propositions 7.18 and 7.21] than alternative recovery maps in quantum information theory [13,Section 4]. 5 In future work, we hope to prove functoriality (without any faithfulness assumptions), continuity, and a complete characterization.…”

“…In 2014, Baez and Fritz provided a categorical Bayesian characterization of the relative entropy of finite probability measures using a category of hypotheses [1]. This was then extended to standard Borel spaces by Gagné and Panangaden in 2018 [5]. An immediate question remains as to whether or not the quantum (Umegaki) relative entropy [12] has a similar characterization.…”

A non-commutative Bayes' theorem

“…We propose an inherently process-theoretic and diagrammatic formulation of quantum Bayesian inference [41]. The approach is based on category theory, which has been providing an interesting perspective on the foundations of probability theory [4,7,9,11,13,16,17,19,21,22,24,31,39], and more recently quantum probability [10,14,23,41] and the information-theoretic foundations of quantum mechanics [6]. The categorical perspective provides a formulation of Bayes' theorem regarding the existence of a certain morphism satisfying a condition equivalent to Bayes' rule in the category of stochastic maps (morphisms are interpreted as a conditional probabilities in this context).…”