Entropy-Based Incremental Variational Bayes Learning of Gaussian Mixtures

Peñalver, Antonio; Escolano, Francisco

doi:10.1109/tnnls.2011.2177670

Cited by 29 publications

(16 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conclude by mentioning that it is still an active research topic to find good GMM learning algorithms in practice (e.g., see the recent entropy-based algorithm [43]).…”

Section: Concluding Remarks and Discussionmentioning

confidence: 99%

K-MLE: A fast algorithm for learning statistical mixture models

Nielsen

2012

2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

We describe k-MLE, a fast and efficient local search algorithm for learning finite statistical mixtures of exponential families such as Gaussian mixture models. Mixture models are traditionally learned using the expectation-maximization (EM) soft clustering technique that monotonically increases the incomplete (expected complete) likelihood. Given prescribed mixture weights, the hard clustering k-MLE algorithm iteratively assigns data to the most likely weighted component and update the component models using Maximum Likelihood Estimators (MLEs). Using the duality between exponential families and Bregman divergences, we prove that the local convergence of the complete likelihood of k-MLE follows directly from the convergence of a dual additively weighted Bregman hard clustering. The inner loop of k-MLE can be implemented using any k-means heuristic like the celebrated Lloyd's batched or Hartigan's greedy swap updates. We then show how to update the mixture weights by minimizing a crossentropy criterion that implies to update weights by taking the relative proportion of cluster points, and reiterate the mixture parameter update and mixture weight update processes until convergence. Hard EM is interpreted as a special case of k-MLE when both the component update and the weight update are performed successively in the inner loop. To initialize k-MLE, we propose k-MLE++, a careful initialization of k-MLE guaranteeing probabilistically a global bound on the best possible complete likelihood.

show abstract

“…We conclude by mentioning that it is still an active research topic to find good GMM learning algorithms in practice (e.g., see the recent entropy-based algorithm [43]).…”

Section: Concluding Remarks and Discussionmentioning

confidence: 99%

K-MLE: A fast algorithm for learning statistical mixture models

Nielsen

2012

2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…When analyzing real datasets, NIST (herein we use 10, 000 samples with d = 86) provides a nice amount of intra-cluster sparsity and inter-cluster noise (both due to ambiguities). We compare our two stage approach (SZE) either applied to the original graph (for a given σ) or to an anchor graph obtained with a nested MDL strategy relying on our EBEM clustering method [3]. In Fig.…”

Section: Experiments With the Nist Datasetmentioning

confidence: 99%

On the Interplay Between Strong Regularity and Graph Densification

Fiorucci

Torcinovich

Curado

et al. 2017

Graph-Based Representations in Pattern Recognition

View full text Add to dashboard Cite

In this paper we analyze the practical implications of Szemerédi's regularity lemma in the preservation of metric information contained in large graphs. To this end, we present a heuristic algorithm to find regular partitions. Our experiments show that this method is quite robust to the natural sparsification of proximity graphs. In addition, this robustness can be enforced by graph densification.The aim of this work is to analyze the ideal density regime where the regularity lemma can find useful applications. In particular, we use the regularity lemma to reduce an input graph and we then exploit the key lemma to obtain an expanded version which preserves some topological properties of the original graph. If we are out of the ideal density regime, we have to densify the graph before applying the regularity lemma. Among the many topological measures we test the effective resistance (or equivalently the scaled commute time), one of the most important metrics between the vertices in the graph, which has been very recently questioned. In [12] it is argued that this measure is meaningless for large graphs. However, recent experimental results show that the graph can be pre-processed (densified) to provide some informative estimation of this metric [5] [4]. Therefore, in this paper, we analyze the practical implications of the key lemma in the estimation of commute time in large graphs.

show abstract

“…Inference for nonparametric models can be conducted under a Bayesian setting, typically by means of variational Bayes (e.g., [20]), or Monte Carlo techniques (e.g., [21]). Here, we prefer a variational Bayesian approach, due to its considerably better scalability in terms of computational costs, which becomes of major importance when having to deal with large data corpora [22], [23]. Our variational Bayesian inference algorithm for the MGPCH model comprises derivation of a family of variational posterior distributions q(.)…”

Section: Inference Algorithmmentioning

confidence: 99%

Gaussian Process-Mixture Conditional Heteroscedasticity

Platanios

Chatzis

2014

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Generalized autoregressive conditional heteroscedasticity (GARCH) models have long been considered as one of the most successful families of approaches for volatility modeling in financial return series. In this paper, we propose an alternative approach based on methodologies widely used in the field of statistical machine learning. Specifically, we propose a novel nonparametric Bayesian mixture of Gaussian process regression models, each component of which models the noise variance process that contaminates the observed data as a separate latent Gaussian process driven by the observed data. This way, we essentially obtain a Gaussian process-mixture conditional heteroscedasticity (GPMCH) model for volatility modeling in financial return series. We impose a nonparametric prior with power-law nature over the distribution of the model mixture components, namely the Pitman-Yor process prior, to allow for better capturing modeled data distributions with heavy tails and skewness. Finally, we provide a copula-based approach for obtaining a predictive posterior for the covariances over the asset returns modeled by means of a postulated GPMCH model. We evaluate the efficacy of our approach in a number of benchmark scenarios, and compare its performance to state-of-the-art methodologies.

show abstract

Entropy-Based Incremental Variational Bayes Learning of Gaussian Mixtures

Cited by 29 publications

References 17 publications

K-MLE: A fast algorithm for learning statistical mixture models

K-MLE: A fast algorithm for learning statistical mixture models

On the Interplay Between Strong Regularity and Graph Densification

Gaussian Process-Mixture Conditional Heteroscedasticity

Contact Info

Product

Resources

About