Parameterizations make different model selections: Empirical findings from factor analysis

Abstract

How parameterizations affect model selection performance is an issue that has been ignored or seldom studied since traditional model selection criteria, such as Akaike's information criterion (AIC), Schwarz's Bayesian information criterion (BIC), difference of negative log-likelihood (DNLL), etc., perform equivalently on different parameterizations that have equivalent likelihood functions. For factor analysis (FA), in addition to one traditional model (shortly denoted by FA-a), it was previously found that there is another parameterization (shortly denoted by FA-b) and the Bayesian YingYang (BYY) harmony learning gets different model selection performances on FA-a and FA-b. This paper investigates a family of FA parameterizations that have equivalent likelihood functions, where each one (shortly denoted by FA-r) is featured by an integer r, with FA-a as one end that r = 0 and FA-b as the other end that r reaches its upper-bound. In addition to the BYY learning in comparison with AIC, BIC, and DNLL, we also implement variational Bayes (VB). Several empirical finds have been obtained via extensive experiments. First, both BYY and VB perform obviously better on FA-b than on FA-a, and this superiority of FA-b is reliable and robust. Second, both BYY and VB outperform AIC, BIC, and DNLL, while BYY further outperforms VB considerably, especially on FA-b. Moreover, with FA-a replaced by FA-b, the gain obtained by BYY is obviously higher than the one by VB, while the gain by VB is better than no gain by AIC, BIC, and DNLL. Third, this paper also demonstrates how each part of priors incrementally and jointly improves the performances, and further shows that using VB to optimize the hyperparameters of priors deteriorates the performances while using BYY for this purpose can further improve the performances.

Reflections on Partial Least Squares Path Modeling

“…Beyond this procedure, there are numerous automated search algorithms that can determine the optimal number of latent variables and system of relations between them (e.g., Landsheer, 2010;Marcoulides & Ing, 2012;Shimizu et al, 2011;Spirtes, Glymour, Scheines, & Tillman, 2010;Tu & Xu, 2011;Xu, 2010Xu, , 2012Zheng & Pavlou, 2010). These techniques will almost invariably return several models that provide a good fit to the data.…”

Further advances on Bayesian Ying-Yang harmony learning

“…• Algorithms and applications see the roadmaps in Figure three (2011) and Sect.5 of Xu (2012a), plus recent applications in (Pang et al 2013;Shi et al 2011aShi et al ,b,c, 2014Tu and Xu 2011a;Tu et al 2011Tu et al , 2012aTu and Xu 2014;Wang et al 2011). …”

“…Such a BYY harmony sparse learning comes from q(A|ρ) that takes a dual role of q(Y |φ). Being different from the existing sparse learning studies (Shi et al 2011a(Shi et al , 2014Tu and Xu 2011a;Xu 2012b) that consider either q(A|ρ) in a long tail distribution with extensive computing cost or q(A|ρ) in Equation 56 with help of one additional q(ρ) (see Sect.III of Xu (2012b) Of course, we may progress to consider a priori q(ρ) and also some priories about , , which will lead to another layer of integral about q(ρ), , . Readers are referred to Sect.2.3 in Xu (2011) .…”

Learning binary factor analysis with automatic model selection

“…(8) and (9), proper priors can be incorporated under a general guideline of BYY learning in [12]. Such efforts have been made on factor analysis in [16], Gaussian mixture model in [17]. The prior term ln qðΘjΞÞ in Eq.…”

“…Such comparisons have been made on factor analysis in [16] and Gaussian mixture model in [17], but not on BFA yet. We simplify the VB-ICA algorithm [18,19] to obtain a VB algorithm on BFA.…”