On the S-instability and degeneracy of discrete deep learning models

Abstract

A probability model exhibits instability if small changes in a data outcome result in large and, often unanticipated, changes in probability. This instability is a property of the probability model, given by a distributional form and a given configuration of parameters. For correlated data structures found in several application areas, there is increasing interest in identifying such sensitivity in model probability structure. We consider the problem of quantifying instability for general probability models defined on sequences of observations, where each sequence of length $N$ has a finite number of possible values that can be taken at each point. A sequence of probability models, indexed by $N$, and an associated parameter sequence result to accommodate data of expanding dimension. Model instability is formally shown to occur when a certain log probability ratio under such models grows faster than $N$. In this case, a one component change in the data sequence can shift probability by orders of magnitude. Also, as instability becomes more extreme, the resulting probability models are shown to tend to degeneracy, placing all their probability on potentially small portions of the sample space. These results on instability apply to large classes of models commonly used in random graphs, network analysis and machine learning contexts.

Properties and Bayesian fitting of restricted Boltzmann machines

“…In other words, S‐unstable RBM model sequences are guaranteed to stack up all probability on a specific set of outcomes for visibles, which potentially could be arbitrarily narrow. Proofs of Propositions and follow from more general results in Kaplan et al . These findings also have counterparts in results in Schweinberger , but unlike results there, are not limited in consideration to exponential family forms with a fixed number of parameters.…”

Modeling and inference for mixtures of simple symmetric exponential families of ‐dimensional distributions for vectors with binary coordinates

“…That is, we consider μ = z j * where j * = arg max{n 1 , n 2 , … , n 2 p }. (10) To develop an estimator for the variability parameter 𝛼, temporarily fix the central pattern 𝝁 in form (9). Let m i be the number of pixel flips that x i is away from 𝝁.…”

“…Note that in the first fraction in display (13) the denominator includes a single term and the numerator is a sum over at most p nonzero terms for a choice of 𝝁. Then in light of forms (10) and (13), one (crude) estimator for 𝛼 is α(μ).…”

“…The fact that multivariate normal mixture modeling itself is closely related to multivariate kernel density estimation makes this approach a sort of "binary vector 'kernel' pmf estimation" methodology. Besides being as flexible (in terms of modeling of an arbitrary distribution for binary vectors) as other modeling approaches such as restricted Boltzmann machines (RBMs) for example, our proposed methodology remains far more interpretable, relatively parsimonious (in comparison to the RBMs), and does not suffer from serious limitations [9,10]. It also provides rational bases for clustering training cases on the basis of posterior probabilities of coming from the same component, a real strength of the conceptualization (especially in where data vectors can be incomplete).…”

On advancing MCMC-based methods for Markovian data structures with applications to deep learning, simulation, and resampling

“…In other words, S-unstable RBM model sequences are guaranteed to stack up all probability on a specific set of outcomes for visibles, which could potentially be arbitrarily narrow. Proofs of Propositions 1 and 2 appear in Kaplan, Nordman, and Vardeman (2017). These findings also have counterparts in results in Schweinberger (2011), but unlike results there, we do not limit consideration to exponential family forms with a fixed number of parameters.…”