Objective priors from maximum entropy in data classification

Palmieri, F.; Ciuonzo, Domenico

doi:10.1016/j.inffus.2012.01.012

Cited by 39 publications

(21 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mérouane Debbah and Ralf Müller correctly describe these joint probabilities as a model with as many degrees of freedom as possible, which leaves free degrees for correlation to exist or not [4] (p.1674). This avoids the introduction of unjustified information [4] (p.1672) corresponding to the simple intuition behind PME: when updating your probabilities, waste no useful information and do not gain information unless the evidence compels you to gain it (see [4] (p.1685f), [5] (p.376), [6,7], [8] (p.186)). The principle comes with its own formal apparatus, not unlike probability theory itself: Shannon's information entropy [9], the Kullback-Leibler divergence (see [10,11], [12] (p.308ff), [13] (p.262ff)), the use of Lagrange multipliers (see [3] (p.409ff), [12] (p.327f), [13] (p.281)), and the log-inverse relationship between information and probability (see [14][15][16][17]).…”

Section: Jeffrey's Updating Principle and The Principle Of Maximum Enmentioning

confidence: 99%

Maximum Entropy and Probability Kinematics Constrained by Conditionals

Lukits

2015

Entropy

View full text Add to dashboard Cite

Two open questions of inductive reasoning are solved: (1) does the principle of maximum entropy (PME) give a solution to the obverse Majerník problem; and (2) is Wagner correct when he claims that Jeffrey's updating principle (JUP) contradicts PME? Majerník shows that PME provides unique and plausible marginal probabilities, given conditional probabilities. The obverse problem posed here is whether PME also provides such conditional probabilities, given certain marginal probabilities. The theorem developed to solve the obverse Majerník problem demonstrates that in the special case introduced by Wagner PME does not contradict JUP, but elegantly generalizes it and offers a more integrated approach to probability updating.

show abstract

Section: Jeffrey's Updating Principle and The Principle Of Maximum Enmentioning

confidence: 99%

Maximum Entropy and Probability Kinematics Constrained by Conditionals

Lukits

2015

Entropy

View full text Add to dashboard Cite

show abstract

“…Process classification based on small values of n may be important in timecritical applications where an inference, or a decision, has to be made as soon as possible. In application where we have knowledge of the likelihoods, but we have no information about priors the classical use of uniform priors may lead to contradictory results [17]. Entropic priors, obtained from maximization of the joint entropy H(X 1:n , S), are to be considered "objective" and are…”

Section: Sequence Classificationmentioning

confidence: 99%

“…Entropic priors are derived from the maximization of model entropy and seem to be an excellent candidate for solving some of the inconsistencies related to model uncertainties [20,19,13,3,4,7,14]. We have presented a discussion on entropic priors in [17] and applied it to target classification [15], to graphical models [5] and to AR process classification [16]. Application of the Maximum Entropy principle [9] to prior determination qualifies entropic priors as a promising alternative to blind use of uniform priors.…”

Section: Introductionmentioning

confidence: 99%

Consistency of sequence classification with entropic priors

Palmieri

Ciuonzo

2012

AIP Conference Proceedings

Self Cite

View full text Add to dashboard Cite

Entropic priors, recently revisited within the context of theoretical physics, were originally introduced for image processing and for general statistical inference. Entropic priors seem to represent a very promising approach to "objective" prior determination when such information is not available. The attention has been mostly limited to continuous parameter spaces and our focus in this work is on the application of the entropic prior idea to Bayesian inference with discrete classes in signal processing problems. Unfortunately, it is well known that entropic priors, when applied to sequences, may lead to excessive spreading of the entropy as the number of samples grows. In this paper we show that the spreading of the entropy may be tolerated if the posterior probabilities remain consistent. We derive a condition based on conditional entropies and KL-divergences for posterior consistency using the Asymptotic Equipartition Property (AEP). Furthermore, we show that entropic priors can be modified to force posterior consistency by adding a constraint to joint entropy maximization. Simulations on the application of entropic priors to a coin flipping experiment are included.

show abstract

“…As a result, the Bayesian and Neyman-Pearson approaches are ineffective due to the absence of complete prior information, and the Minimax approach achieves a poor performance since the available partial prior information is ignored. In order to utilize the partial information and to achieve a better performance, several studies have been conducted [8,[17][18][19][20][21][22][23][24][25], for example, maximum entropy (ME), Γ-minimax, restricted Bayes, and restricted Neyman-Pearson approaches, to name but a few. For instance, the ME method is utilized in [23] to translate the information contained in the known form of the likelihood into a prior distribution for Bayesian inference.…”

Section: Introductionmentioning

confidence: 99%

Optimal Detection under the Restricted Bayesian Criterion

Liu

Yang

Liu

2017

Entropy

View full text Add to dashboard Cite

This paper aims to find a suitable decision rule for a binary composite hypothesis-testing problem with a partial or coarse prior distribution. To alleviate the negative impact of the information uncertainty, a constraint is considered that the maximum conditional risk cannot be greater than a predefined value. Therefore, the objective of this paper becomes to find the optimal decision rule to minimize the Bayes risk under the constraint. By applying the Lagrange duality, the constrained optimization problem is transformed to an unconstrained optimization problem. In doing so, the restricted Bayesian decision rule is obtained as a classical Bayesian decision rule corresponding to a modified prior distribution. Based on this transformation, the optimal restricted Bayesian decision rule is analyzed and the corresponding algorithm is developed. Furthermore, the relation between the Bayes risk and the predefined value of the constraint is also discussed. The Bayes risk obtained via the restricted Bayesian decision rule is a strictly decreasing and convex function of the constraint on the maximum conditional risk. Finally, the numerical results including a detection example are presented and agree with the theoretical results.

show abstract

Objective priors from maximum entropy in data classification

Cited by 39 publications

References 28 publications

Maximum Entropy and Probability Kinematics Constrained by Conditionals

Maximum Entropy and Probability Kinematics Constrained by Conditionals

Consistency of sequence classification with entropic priors

Optimal Detection under the Restricted Bayesian Criterion

Contact Info

Product

Resources

About