Identifiability of parameters in latent structure models with many observed variables

Abstract

While hidden class models of various types arise in many statistical applications, it is often difficult to establish the identifiability of their parameters. Focusing on models in which there is some structure of independence of some of the observed variables conditioned on hidden ones, we demonstrate a general approach for establishing identifiability utilizing algebraic arguments. A theorem of J. Kruskal for a simple latent-class model with finite state space lies at the core of our results, though we apply it to a diverse set of models. These include mixtures of both finite and nonparametric product distributions, hidden Markov models and random graph mixture models, and lead to a number of new results and improvements to old ones.In the parametric setting, this approach indicates that for such models, the classical definition of identifiability is typically too strong. Instead generic identifiability holds, which implies that the set of nonidentifiable parameters has measure zero, so that parameter inference is still meaningful. In particular, this sheds light on the properties of finite mixtures of Bernoulli products, which have been used for decades despite being known to have nonidentifiable parameters. In the nonparametric setting, we again obtain identifiability only when certain restrictions are placed on the distributions that are mixed, but we explicitly describe the conditions.

Nonparametric spectral-based estimation of latent structures

“…More recently, Hall and Zhou [2003] showed that mixtures of two arbitary distributions are generally identified as soon as three measurements are available, provided the component distributions are linearly independent and the outcomes satisfy a conditional-independence restriction. Allman, Matias and Rhodes [2009] have demonstrated that this result carries over to mixtures of more components. Their approach can be applied to a more general class of latent structures that feature some form of conditional independence, such as hidden Markov models with finite state spaces (see Petrie [1969] for seminal work on this) and to random-graph models.…”

“…However, in contrast to Allman, Matias and Rhodes [2009], we go beyond appealing to his results to claim identification of the decomposition. Moreover, we show that a simple transformation of the array leads to a set of multilinear restrictions that identify the parameters of the latent structure at hand in a constructive manner.…”

“…The work of Allman, Matias and Rhodes [2009] builds heavily on algebraic results for multiway arrays due to Kruskal [1976;1977]. Although widely applicable, this approach is not constructive.…”

“…We next introduce a convenient estimator, discuss its implementation, and provide distribution theory. Our framework is the same as that of Allman, Matias and Rhodes [2009] and, as such, can be used to construct estimators for all structures mentioned above. In particular, we discuss the nonparametric estimation of multivariate finite mixtures of discrete and continuous measures, and of hidden Markov models as illustrations of our proposal.…”

Parameter Identifiability of Discrete Bayesian Networks with Hidden Variables

“…Thus in answering generic identifiability questions one need only consider Markov equivalence classes of DAGs. In Section 4 we revisit the fundamental result due to Kruskal [13], as developed in Allman et al [10] for identifiability questions. We give explicit identifiability procedures for the DAG this to which this result applies most directly (model 3-0), and also for the DAG of model 4-3b.…”