Layered Dynamic Mixture Model for Pattern Discovery in Asynchronous Multi-modal Streams

Abstract

We propose a layered dynamic mixture model for asynchronous multi-modal fusion for unsupervised pattern discovery in video. The lower layer of the model uses generative temporal structures such as a hierarchical hidden Markov model to convert the audio-visual streams into mid-level labels, it also models the correlations in text with probabilistic latent semantic analysis. The upper layer fuses the statistical evidence across diverse modalities with a flexible meta-mixture model that assumes loose temporal correspondence. Evaluation on a large news database shows that multi-modal clusters have better correspondence to news topics than audiovisual clusters alone; novel analysis techniques suggest that meaningful clusters occur when the prediction of salient features by the model concurs with those shown in the story clusters. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved.

Dynamic Adaptive Mixture Models with an Application to Volatility and Risk

“…Yu (2012). Dynamic Mixture Models have been successfully applied in process monitoring (Yu, 2012), intervention detections (Gerlach et al, 2000), insurance losses (Frigessi et al, 2002), and graphical engineering (KaewTraKulPong and Bowden, 2002;Xie et al, 2005). A drawback of these models is that, when nonlinear non-Gaussian specifications are assumed for the mixture components and for the evolution of the mixture composition, classical inference cannot be applied anymore, see e.g.…”