Mismatched estimation and relative entropy

Verdú, S.

doi:10.1109/isit.2009.5205651

Cited by 35 publications

(54 citation statements)

References 30 publications

(53 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This bias is a type of random error, as opposed to a systematic error. Any bias ensures that the minimum error is not achieved for the wrong model, and borrowing a term from financial decision theory 27 and signal processing, 28 we shall refer to the positive deviation from the minimum error as regret .…”

Section: Theorymentioning

confidence: 99%

One Size Does Not Fit All: The Limits of Structure-Based Models in Drug Discovery

Ross

Morris

Biggin

2013

J. Chem. Theory Comput.

View full text Add to dashboard Cite

A major goal in computational chemistry has been to discover the set of rules that can accurately predict the binding affinity of any protein-drug complex, using only a single snapshot of its three-dimensional structure. Despite the continual development of structure-based models, predictive accuracy remains low, and the fundamental factors that inhibit the inference of all-encompassing rules have yet to be fully explored. Using statistical learning theory and information theory, here we prove that even the very best generalized structure-based model is inherently limited in its accuracy, and protein-specific models are always likely to be better. Our results refute the prevailing assumption that large data sets and advanced machine learning techniques will yield accurate, universally applicable models. We anticipate that the results will aid the development of more robust virtual screening strategies and scoring function error estimations.

show abstract

Section: Theorymentioning

confidence: 99%

One Size Does Not Fit All: The Limits of Structure-Based Models in Drug Discovery

Ross

Morris

Biggin

2013

J. Chem. Theory Comput.

View full text Add to dashboard Cite

show abstract

“…Using the identity defined in (7) and the definition of entropy defined above, the mutual information I(X; Y ) becomes…”

Section: B Vector Poisson Channelmentioning

confidence: 99%

“…Second, scaling the support of a Poisson random variable (integers) does not result in another Poisson random variable. This is in contrast to Gaussian channels which are relatively well-studied from both information-theoretic and estimation-theoretic aspects [5], [6], [7]. The Poisson channel has the advantage in thought experiments such as ours of simple conceptual models for data acquisition based on counting (photons, say) and switches that either allow counts to either pass through or be discarded.…”

Section: Introductionmentioning

confidence: 99%

Untitled

2021

SIPIJ

View full text Add to dashboard Cite

It is an experimental design problem in which there are two Poisson sources with two possible and known rates, and one counter. Through a switch, the counter can observe the sources individually or the counts can be combined so that the counter observes the sum of the two. The sensor scheduling problem is to determine an optimal proportion of the available time to be allocated toward individual and joint sensing, under a total time constraint. Two different metrics are used for optimization: mutual information between the sources and the observed counts, and probability of detection for the associated source detection problem. Our results, which are primarily computational, indicate similar but not identical results under the two cost functions.

show abstract

“…This connection was made in the work of Guo, Shamai, and Verdú in [3]. Extensions of the I-MMSE relation were investigated in [8][9][10][11][12][13][14][15][16], and applications have been established, e.g., in optimal power allocation [17] and monotonicity of non-Gaussianness [18]. Our work is inscribed within this literature.…”

Section: 2 Related Literaturementioning

confidence: 99%

Measuring Information from Moments

Alghamdi¹,

Calmon²

2021

Preprint

View full text Add to dashboard Cite

We investigate the problem of representing information measures in terms of the moments of the underlying random variables. First, we derive polynomial approximations of the conditional expectation operator. We then apply these approximations to bound the best mean-square error achieved by a polynomial estimator-referred to here as the PMMSE. In Gaussian channels, the PMMSE coincides with the minimum mean-square error (MMSE) if and only if the input is either Gaussian or constant, i.e., if and only if the conditional expectation of the input of the channel given the output is a polynomial of degree at most 1. By combining the PMMSE with the I-MMSE relationship, we derive new formulas for information measures (e.g., differential entropy, mutual information) that are given in terms of the moments of the underlying random variables. As an application, we introduce estimators for information measures from data via approximating the moments in our formulas by sample moments. These estimators are shown to be asymptotically consistent and possess desirable properties, e.g., invariance to affine transformations when used to estimate mutual information.

show abstract

Mismatched estimation and relative entropy

Cited by 35 publications

References 30 publications

One Size Does Not Fit All: The Limits of Structure-Based Models in Drug Discovery

One Size Does Not Fit All: The Limits of Structure-Based Models in Drug Discovery

Untitled

Measuring Information from Moments

Contact Info

Product

Resources

About