Field Theories for Learning Probability Distributions

Bialek, William; Callan, Curtis G.; Strong, S. P.

doi:10.1103/physrevlett.77.4693

Cited by 102 publications

(153 citation statements)

References 3 publications

Supporting

Mentioning

152

Contrasting

Order By: Relevance

“…However, an "optimal" value for the smoothness controlling parameter was derived from the data itself, a topic also addressed by Stoica et al [46] and by a follow up publication to ours [47]. Bialek et al [45] also recognized, as we do, that an IFT can easily be non-local.…”

Section: Statistical and Bayesian Field Theorymentioning

confidence: 93%

“…Bialek et al [45] applied a field theoretical approach to recover a probability distribution from data. Here, a Bayesian prior was used to regularize the solution, which was set up ad-hoc to enforce smoothness of the reconstruction, obtained from the classical (or saddlepoint, or maximum a posteriori) solution of the problem.…”

Section: Statistical and Bayesian Field Theorymentioning

confidence: 99%

See 1 more Smart Citation

Information field theory for cosmological perturbation reconstruction and nonlinear signal analysis

2009

View full text Add to dashboard Cite

We develop information field theory (IFT) as a means of Bayesian inference on spatially distributed signals, the information fields. A didactical approach is attempted. Starting from general considerations on the nature of measurements, signals, noise, and their relation to a physical reality, we derive the information Hamiltonian, the source field, propagator, and interaction terms. Free IFT reproduces the well known Wiener-filter theory. Interacting IFT can be diagrammatically expanded, for which we provide the Feynman rules in position-, Fourier-, and spherical harmonics space, and the Boltzmann-Shannon information measure. The theory should be applicable in many fields. However, here, two cosmological signal recovery problems are discussed in their IFTformulation. 1) Reconstruction of the cosmic large-scale structure matter distribution from discrete galaxy counts in incomplete galaxy surveys within a simple model of galaxy formation. We show that a Gaussian signal, which should resemble the initial density perturbations of the Universe, observed with a strongly non-linear, incomplete and Poissonian-noise affected response, as the processes of structure and galaxy formation and observations provide, can be reconstructed thanks to the virtue of a response-renormalization flow equation. 2) We design a filter to detect local non-linearities in the cosmic microwave background, which are predicted from some Early-Universe inflationary scenarios, and expected due to measurement imperfections. This filter is the optimal Bayes' estimator up to linear order in the non-linearity parameter and can be used even to construct sky maps of non-linearities in the data.

show abstract

Section: Statistical and Bayesian Field Theorymentioning

confidence: 93%

Section: Statistical and Bayesian Field Theorymentioning

confidence: 99%

Information field theory for cosmological perturbation reconstruction and nonlinear signal analysis

2009

View full text Add to dashboard Cite

show abstract

“…It encodes a priori assumptions about the typical variability of model functions f with the input x. Such Gaussian process models (GP) have attracted considerable attention in recent years as they represent a flexible and widely applicable concept [4,5,6,7]. GP models can be understood as a limit of Bayesian feed-forward neural networks when the number of hidden units grows to infinity [5].…”

mentioning

confidence: 99%

Statistical Mechanics of Learning: A Variational Approach for Real Data

Malzahn¹,

Opper²

2002

Phys. Rev. Lett.

View full text Add to dashboard Cite

Using a variational technique, we generalize the statistical physics approach of learning from random examples to make it applicable to real data. We demonstrate the validity and relevance of our method by computing approximate estimators for generalization errors that are based on training data alone. PACS numbers: 84.35.+i,87.18.Sn In recent years, methods of statistical physics have contributed important insights to the theory of learning from example data with neural networks and other learning machines (for recent reviews see e.g. [1,2]). Such systems are often described by models with a large number of degrees of freedom which interact by a random energy function where the randomness in a learning problem is induced by the data. Exact solutions for the learning performance of a great variety of models have been obtained using tools which were developed for the physics of disordered materials, such as the replica trick [3]. All these case studies assume simple distributions of the data and give invaluable insights into the generic learning behavior. However, they do not (and did not intend to) describe real world learning experiments where one has the additional problem that the distribution of the data is often unknown and idealized assumptions about it seem to be too restrictive. It would be highly desirable if the statistical physics theory could provide practitioners with tools to predict and optimize the performance of a learning algorithm.In this Letter, we will present a step forward in this direction. We combine the replica approach with a variational approximation which enables us to deal with more realistic distributions of the randomness. For models of supervised learning [1], we demonstrate that the theory can predict relations between experimentally measurable quantities even though details of the data distribution are unknown. As an application, we will derive approximate expressions for data averaged performance measures for state of the art learning algorithms and test them on real data sets. We hope that our approach will inspire similar research in related complex adaptive systems such as, for example, communication systems [2].We demonstrate the basic idea of our approach on a typical learning scenario where we try to learn an unknown rule which maps inputs x ∈ R d to outputs y from a set D of m example data pairs (x i , y i ), i = 1, . . . , m. The possible rules are modeled by a class of real valued functions f (x). To get a reasonable predictorf (x|D) on arbitrary novel inputs x, a popular learning strategy is to balance the goodness of fit on the training data measured by a training energy, y i ) with the prior knowledge about the complexity of rules. In a probabilistic, Bayesian approach to learning [1,2], both ingredients are combined in a probability distribution

show abstract

“…There it has also been suggested that the decay of the estimation error with the number of available data points can be made largely independent of model mismatch by optimal hyperparameter tuning [18,19,20,21,22,23,24,25].…”

Section: Resultsmentioning

confidence: 99%

Can Gaussian Process Regression Be Made Robust Against Model Mismatch?

Sollich

2005

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Learning curves for Gaussian process (GP) regression can be strongly affected by a mismatch between the 'student' model and the 'teacher' (true data generation process), exhibiting e.g. multiple overfitting maxima and logarithmically slow learning. I investigate whether GPs can be made robust against such effects by adapting student model hyperparameters to maximize the evidence (data likelihood). An approximation for the average evidence is derived and used to predict the optimal hyperparameter values and the resulting generalization error. For large input space dimension, where the approximation becomes exact, Bayes-optimal performance is obtained at the evidence maximum, but the actual hyperparameters (e.g. the noise level) do not necessarily reflect the properties of the teacher. Also, the theoretically achievable evidence maximum cannot always be reached with the chosen set of hyperparameters, and maximizing the evidence in such cases can actually make generalization performance worse rather than better. In lower-dimensional learning scenarios, the theory predicts-in excellent qualitative and good quantitative accord with simulations-that evidence maximization eliminates logarithmically slow learning and recovers the optimal scaling of the decrease of generalization error with training set size.

show abstract

Field Theories for Learning Probability Distributions

Cited by 102 publications

References 3 publications

Information field theory for cosmological perturbation reconstruction and nonlinear signal analysis

Information field theory for cosmological perturbation reconstruction and nonlinear signal analysis

Statistical Mechanics of Learning: A Variational Approach for Real Data

Can Gaussian Process Regression Be Made Robust Against Model Mismatch?

Contact Info

Product

Resources

About