Convex foundations for generalized MaxEnt models

Frongillo, Rafael M.; Reid, Mark D.

doi:10.1063/1.4903703

Cited by 6 publications

(15 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The form of ρ t is to be compared to the generalized exponential family in [24] and the generalized MaxEnt models of [21].…”

Section: Comparison Of the Bound On An Example In The Continuous Casementioning

confidence: 99%

Non-exponentially weighted aggregation: regret bounds for unbounded loss functions

Alquier¹

2020

Preprint

View full text Add to dashboard Cite

We tackle the problem of online optimization with a general, possibly unbounded, loss function. It is well known that the exponentially weighted aggregation strategy (EWA) leads to a regret in √ T after T steps, under the assumption that the loss is bounded. The online gradient algorithm (OGA) has a regret in √ T when the loss is convex and Lipschitz. In this paper, we study a generalized aggregation strategy, where the weights do no longer necessarily depend exponentially on the losses. Our strategy can be interpreted as the minimization of the expected losses plus a penalty term. When the penalty term is the Kullback-Leibler divergence, we obtain EWA as a special case, but using alternative divergences lead to a regret bounds for unbounded, not necessarily convex losses. However, the cost is a worst regret bound in some cases.

show abstract

“…The form of ρ t is to be compared to the generalized exponential family in [24] and the generalized MaxEnt models of [21].…”

Section: Comparison Of the Bound On An Example In The Continuous Casementioning

confidence: 99%

Non-exponentially weighted aggregation: regret bounds for unbounded loss functions

Alquier¹

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…where θ ∈ R d is a vector of (possibly negative) prediction scores produced by a model f W (x) and p ∈ d is a discrete probability distribution. It is a generalized exponential family distribution (Grünwald and Dawid, 2004;Frongillo and Reid, 2014) with natural parameter θ ∈ R d and regularization Ω. Of particular interest is the case where y Ω (θ) is sparse, meaning that there are scores θ for which the resulting y Ω (θ) assigns zero probability to some classes.…”

Section: Probabilistic Prediction With Fenchel-young Lossesmentioning

confidence: 99%

“…In this section, we discuss the concept of probability distribution regularization. Our treatment follows closely the variational formulations of exponential families (Barndorff-Nielsen, 1978;Wainwright and Jordan, 2008) and generalized exponential families (Grünwald and Dawid, 2004;Frongillo and Reid, 2014) but adopts the novel viewpoint of regularized prediction functions. We discuss two instances of that framework: the structured counterpart of the softmax, marginal inference, and a new structured counterpart of the sparsemax, structured sparsemax.…”

Section: Distribution Regularization Marginal Inference and Structure...mentioning

confidence: 99%

Learning with Fenchel-Young Losses

Blondel¹,

Martins²,

Niculae³

2019

Preprint

View full text Add to dashboard Cite

Over the past decades, numerous loss functions have been been proposed for a variety of supervised learning tasks, including regression, classification, ranking, and more generally structured prediction. Understanding the core principles and theoretical properties underpinning these losses is key to choose the right loss for the right problem, as well as to create new losses which combine their strengths. In this paper, we introduce Fenchel-Young losses, a generic way to construct a convex loss function for a regularized prediction function. We provide an in-depth study of their properties in a very broad setting, covering all the aforementioned supervised learning tasks, and revealing new connections between sparsity, generalized entropies, and separation margins. We show that Fenchel-Young losses unify many well-known loss functions and allow to create useful new ones easily. Finally, we derive efficient predictive and training algorithms, making Fenchel-Young losses appealing both in theory and practice.

show abstract

“…In fact, Theorem 2 is so important that we state and prove a generalization of it in appendix, Section 9, showing that dropping the "same family" constraint does not change the f -divergence (information-theoretic) vs Bregman divergence (information-geometric) picture. We now define generalizations of exponential families, following [6,23]. Let χ : R + → R + be non-decreasing [41,Chapter 10].…”

Section: Definitionsmentioning

confidence: 99%

“…To our knowledge, there is no previously known "GAN-amenable" generalization of this identity above exponential families. Related identities have recently been proven for two generalizations of exponential families [6,Theorem 9], [23,Theorem 3], but fall short of the f -divergence formulation and are not amenable to the variational GAN formulation.…”

Section: Introductionmentioning

confidence: 99%

f-GANs in an Information Geometric Nutshell

Nock¹,

Cranko²,

Menon³

et al. 2017

Preprint

View full text Add to dashboard Cite

Nowozin et al showed last year how to extend the GAN principle to all f -divergences. The approach is elegant but falls short of a full description of the supervised game, and says little about the key player, the generator: for example, what does the generator actually converge to if solving the GAN game means convergence in some space of parameters? How does that provide hints on the generator's design and compare to the flourishing but almost exclusively experimental literature on the subject?In this paper, we unveil a broad class of distributions for which such convergence happens -namely, deformed exponential families, a wide superset of exponential families -and show tight connections with the three other key GAN parameters: loss, game and architecture. In particular, we show that current deep architectures are able to factorize a very large number of such densities using an especially compact design, hence displaying the power of deep architectures and their concinnity in the f -GAN game. This result holds given a sufficient condition on activation functions -which turns out to be satisfied by popular choices. The key to our results is a variational generalization of an old theorem that relates the KL divergence between regular exponential families and divergences between their natural parameters. We complete this picture with additional results and experimental insights on how these results may be used to ground further improvements of GAN architectures, via (i) a principled design of the activation functions in the generator and (ii) an explicit integration of proper composite losses' link function in the discriminator.

show abstract

Convex foundations for generalized MaxEnt models

Cited by 6 publications

References 17 publications

Non-exponentially weighted aggregation: regret bounds for unbounded loss functions

Non-exponentially weighted aggregation: regret bounds for unbounded loss functions

Learning with Fenchel-Young Losses

f-GANs in an Information Geometric Nutshell

Contact Info

Product

Resources

About