Gibbs ensembles for incompatible dependency networks

Abstract

In most statistical applications, the Gibbs sampler is the method of choice for inference regarding conditionally specified distributions that are compatible. Compatibility ensures that a unique Gibbs distribution exists. For machine learning of complex models such as dependency networks, the conditional models are sometimes incompatible. In this paper, we review an ensemble approach using the Gibbs sampler as the base procedure. A Gibbs ensemble consists of many joint distributions resulting from different scan orders of the same conditional model, and the solution is a weighted sum of the ensemble. The algorithm is scalable and can handle large data sets of high dimensionality. The proposed approach provides joint distributions that conform with the conditional specifications better than the solutions obtained by linear programming and by a fixed-scan Gibbs sampler alone. Owing to incompatibility, the invariant distribution of a Gibbs sampler is scan-order dependent. A Gibbs ensemble is the collection of joint distributions estimated from the Gibbs samples of different scan orders.

System-Subsystem Dependency Network for Integrating Multicomponent Data and Its Application to Health Sciences

“…Fixed-scan Gibbs samplers follow a prefixed order of sampling-e.g., for 3 variables, X 1 → X 2 → X 3 → X 1 and so on [3]. However, the fixed-scan pseudo-Gibbs sampler, which is commonly used in Bayesian estimation, is not optimal for potentially incompatible conditional distributions in the sense that the conditional distributions derived from the inferred joint distribution may have large error variances [4,5]. It has been shown that by exploiting the full range of possible scan orders, either through a technique called the Gibbs ensemble or through randomizing the scan order during sampling [15] in the pseudo-Gibbs procedure, one can substantially reduce the level of incompatibility measured in error variances.…”