Bayesian network structure learning using factorized NML universal models

Roos, Teemu; Silander, Tomi; Kontkanen, Petri; Myllymäki, Petri

doi:10.1109/ita.2008.4601061

Cited by 24 publications

(11 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Note that this pNML probability assignment was essentially proposed earlier, see [7], [8], with a different motivation as one of the possible variations of the Normalized Maximum Likelihood (NML) method of [6] for universal prediction.…”

Section: Introductionmentioning

confidence: 99%

A New Look at an Old Problem: A Universal Learning Approach to Linear Regression

Bibas

Fogel

Feder

2019

2019 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

Linear regression is a classical paradigm in statistics. A new look at it is provided via the lens of universal learning. In applying universal learning to linear regression the hypotheses class represents the label y ∈ R as a linear combination of the feature vector x T θ where x ∈ R M , within a Gaussian error. The Predictive Normalized Maximum Likelihood (pNML) solution for universal learning of individual data can be expressed analytically in this case, as well as its associated learnability measure. Interestingly, the situation where the number of parameters M may even be larger than the number of training samples N can be examined. As expected, in this case learnability cannot be attained in every situation; nevertheless, if the test vector resides mostly in a subspace spanned by the eigenvectors associated with the large eigenvalues of the empirical correlation matrix of the training data, linear regression can generalize despite the fact that it uses an "over-parametrized" model. We demonstrate the results with a simulation of fitting a polynomial to data with a possibly large polynomial degree.

show abstract

Section: Introductionmentioning

confidence: 99%

A New Look at an Old Problem: A Universal Learning Approach to Linear Regression

Bibas

Fogel

Feder

2019

2019 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

show abstract

“…Even though it is strongly principled, NML computation is restricted to certain classes of models, e.g, multinomial distribution (Kontkanen & Myllymäki, 2007), naive Bayes (Mononen & Myllymäki, 2007), which prevents its use in score-based structure learning. In Bayesian networks, efficient approximations were proposed and shown to perform better in model selection (Roos et al, 2008;Silander et al, 2018).…”

Section: Nml Regret Estimationmentioning

confidence: 99%

Batch Bayesian Optimization on Permutations using the Acquisition Weighted Kernel

Oh¹,

Bondesan²,

Gavves³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this work we propose a batch Bayesian optimization method for combinatorial problems on permutations, which is well suited for expensive cost functions on permutations. We introduce LAW, a new efficient batch acquisition method based on the determinantal point process, using an acquisition weighted kernel. Relying on multiple parallel evaluations, LAW accelerates the search for the optimal permutation. We provide a regret analysis for our method to gain insight in its theoretical properties. We then apply the framework to permutation problems, which have so far received little attention in the Bayesian Optimization literature, despite their practical importance. We call this method LAW2ORDER. We evaluate the method on several standard combinatorial problems involving permutations such as quadratic assignment, flowshop scheduling and the traveling salesman, as well as on a structure learning task.

show abstract

“…where k 0 P;Q ðNÞ corresponds to a complexity term introduced in [14,15] to discriminate between variable dependence (for I 0 N ð½X� P ; ½Y� Q Þ > 0) and variable independence (for I 0 N ð½X� P ; ½Y� Q Þ⩽0) given a finite dataset of size N. In the present context of finding an optimum discretization for continuous variables, this complexity term introduces a penalty which eventually outweights the information gain in refining bin partitions further, when there is not enough data to support such a refined model, as depicted on Fig 1. For discrete variables, typical complexity terms correspond to the Bayesian Information Criterion (BIC), k BIC P;Q ðNÞ ¼ 1=2ðr x À 1Þðr y À 1Þ log N, where r x and r y are the number of bins for X and Y, or the X-and Y-Normalized Maximum Likelihood (NML) criteria [14][15][16], defined as,…”

Section: Assessing Information In Continuous or Mixed-type Datamentioning

confidence: 99%

Learning clinical networks from medical records based on information estimates in mixed-type data

et al. 2020

View full text Add to dashboard Cite

The precise diagnostics of complex diseases require to integrate a large amount of information from heterogeneous clinical and biomedical data, whose direct and indirect interdependences are notoriously difficult to assess. To this end, we propose an efficient computational approach to simultaneously compute and assess the significance of multivariate information between any combination of mixed-type (continuous/categorical) variables. The method is then used to uncover direct, indirect and possibly causal relationships between mixed-type data from medical records, by extending a recent machine learning method to reconstruct graphical models beyond simple categorical datasets. The method is shown to outperform existing tools on benchmark mixed-type datasets, before being applied to analyze the medical records of eldery patients with cognitive disorders from La Pitié-Salpêtrière Hospital, Paris. The resulting clinical network visually captures the global interdependences in these medical records and some facets of clinical diagnosis practice, without specific hypothesis nor prior knowledge on any clinically relevant information. In particular, it provides some physiological insights linking the consequence of cerebrovascular accidents to the atrophy of important brain structures associated to cognitive impairment.

show abstract

Bayesian network structure learning using factorized NML universal models

Cited by 24 publications

References 19 publications

A New Look at an Old Problem: A Universal Learning Approach to Linear Regression

A New Look at an Old Problem: A Universal Learning Approach to Linear Regression

Batch Bayesian Optimization on Permutations using the Acquisition Weighted Kernel

Learning clinical networks from medical records based on information estimates in mixed-type data

Contact Info

Product

Resources

About