Why Some Families of Probability Distributions Are Practically Efficient: A Symmetry-Based Explanation

Abstract

Out of many possible families of probability distributions, some families turned out to be most efficient in practical situations. Why these particular families and not others? To explain this empirical success, we formulate the general problem of selecting a distribution with the largest possible utility under appropriate constraints. We then show that if we select the utility functional and the constraints which are invariant under natural symmetries -shift and scaling corresponding to changing the starting point and the measuring unit for describing the corresponding quantity x. then the resulting optimal families of probability distributions indeed include most of the empirically successful families. Thus, we get a symmetry-based explanation for their empirical success.

Why Student Distributions? Why Matern’s Covariance Model? A Symmetry-Based Explanation

“…It turns out that entropy is not the only functional with the above scale-invariance properties. All such scale-invariant functions have been described [5,6]. In addition to entropy, we can also have ∫ ln(ρ(x)) dx and ∫ (ρ(x)) q dx for some q.…”

What If We Do Not Know Correlations?

“…It turns out that entropy is not the only functional with the above scale-invariance properties. All such scale-invariant functions have been described [5,6]. In addition to entropy, we can also have For scale-invariant generalizations of entropy, we get Student distribution.…”

Robustness as a Criterion for Selecting a Probability Distribution Under Uncertainty

“…Traditionally in probability theory, when we only have partial knowledge about the probability distribution, we select a distribution for which the entropy − ∫ ρ(x) · ln(ρ(x)) dx attains the largest possible value (see, e.g., [3]), or, equivalently, for which the integral ∫ ρ(x) · ln(ρ(x)) dx attains the smallest possible value. It is worth mentioning that, in general, if we assume that the criterion for selecting a probability distribution is scale-invariant (in some reasonable sense), then this criterion is equivalent to optimizing either entropy, or generalized entropy ∫ ln(ρ(x)) dx or ∫ ρ α (x) dx, for some α > 0; see, e.g., [5]. Our analysis shows that the generalized entropy corresponding to α = 2 and α = 3 describes mean-squared robustness.…”