A Comparative Investigation on Model Selection in Independent Factor Analysis

Abstract

Abstract. With uncorrelated Gaussian factors extended to mutually independent factors beyond Gaussian, the conventional factor analysis is extended to what is recently called independent factor analysis. Typically, it is called binary factor analysis (BFA) when the factors are binary and called non-Gaussian factor analysis (NFA) when the factors are from real non-Gaussian distributions. A crucial issue in both BFA and NFA is the determination of the number of factors. In the literature of statistics, there are a number of model selection criteria that can be used for this purpose. Also, the Bayesian Ying-Yang (BYY) harmony learning provides a new principle for this purpose. This paper further investigates BYY harmony learning in comparison with existing typical criteria, including Akaik_s information criterion (AIC), the consistent Akaike_s information criterion (CAIC), the Bayesian inference criterion (BIC), and the cross-validation (CV) criterion on selection of the number of factors. This comparative study is made via experiments on the data sets with different sample sizes, data space dimensions, noise variances, and hidden factors numbers. Experiments have shown that for both BFA and NFA, in most cases BIC outperforms AIC, CAIC, and CV while the BYY criterion is either comparable with or better than BIC. In consideration of the fact that the selection by these criteria has to be implemented at the second stage based on a set of candidate models which have to be obtained at the first stage of parameter learning, while BYY harmony learning can provide not only a new class of criteria implemented in a similar way but also a new family of algorithms that perform parameter learning at the first stage with automated model selection, BYY harmony learning is more preferred since computing costs can be saved significantly. (2000): 68Q10, 62H25, 68T05, 65C10, 68Q32. Mathematical Subject Classifications

Transcription Network Analysis by A Sparse Binary Factor Analysis Algorithm

“…5 of [24] and Sect IV(C) in [15], at the special case of a two-component Gaussian mixture by Eq. (5), or the BYY BFA learning given on p840 of [15] and also the one studied in [19,21]. The other three BYY-BFA algorithms extends these previous studies by considering either or both of a priori connectivity and a priori q(Θ).…”

“…Research on BFA has been conducted with wide applications, on analysis of binary data (e.g., social research questionnaires, market basket data, etc.) with the aid of Boolean algebra [17], or to discover binary factors in continuous data, e.g., see page 839-840 of [18] and also see [19,20,21].…”

“…One natural way to model a binary variable y j in Eq. (4) is Bernoulli distribution [18,19,21]. Equivalently, we consider that y takes binary values from a two-component Gaussian mixture, that is,…”

Learning binary factor analysis with automatic model selection

“…Unlike the conventional factor analysis where the latent factor is assumed to be Gaussian, BFA traces the observation to independent Bernoulli information sources. Research on BFA has been focused on analysis of binary data, such as social research questionnaires and market basket data, with the aid of Boolean algebra [1], and also on the discovery of binary factors in continuous data, [2][3][4], taking advantage of the representational capacity of the underlying binary structure. When considering all the random variables to be binary, factor analysis becomes the restricted Boltzmann machine which is the building block of the deep belief network [5].…”

“…When considering all the random variables to be binary, factor analysis becomes the restricted Boltzmann machine which is the building block of the deep belief network [5]. This paper considers the same BFA model as in [4,2], under Bayesian Ying-Yang (BYY) harmony learning [6,7], in a comparison with Variation Bayes (VB) [8] and Bayesian information criterion (BIC) [9]. Rissanen's Minimum Description Length (MDL) stems from another viewpoint but coincides with BIC when it is simplified to a simple computable criterion [10].…”