An investigation of several typical model selection criteria for detecting the number of signals

Abstract

Based on the problem of detecting the number of signals, this paper provides a systematic empirical investigation on model selection performances of several classical criteria and recently developed methods (including Akaike's information criterion (AIC), Schwarz's Bayesian information criterion, Bozdogan's consistent AIC, Hannan-Quinn information criterion, Minka's (MK) principal component analysis (PCA) criterion, Kritchman & Nadler's hypothesis tests (KN), Perry & Wolfe's minimax rank estimation thresholding algorithm (MM), and Bayesian Ying-Yang (BYY) harmony learning), by varying signal-to-noise ratio (SNR) and training sample size N . A family of model selection indifference curves is defined by the contour lines of model selection accuracies, such that we can examine the joint effect of N and SNR rather than merely the effect of either of SNR and N with the other fixed as usually done in the literature. The indifference curves visually reveal that all methods demonstrate relative advantages obviously within a region of moderate N and SNR. Moreover, the importance of studying this region is also confirmed by an alternative reference criterion by maximizing the testing likelihood. It has been shown via extensive simulations that AIC and BYY harmony learning, as well as MK, KN, and MM, are relatively more robust than the others against decreasing N and SNR, and BYY is superior for a small sample size.

A new multivariate test formulation: theory, implementation, and applications to genome-scale sequencing and expression

“…FA-b becomes more favourable for determining m. Readers are referred to Sect.2.2 in Xu (2011) and Tu and Xu (2011) for further studies on FA-b versus FA-a.…”

Parameterizations make different model selections: Empirical findings from factor analysis

“…[31] for the performance evaluation on the three levels of implementations of VB and BYY for FA-a and FA-b, and also with the performances on AIC, BIC and DNLL included for comparisons.…”

“…2 to 4 through the contour maps suggested in Ref [31]. for illustrating the joint effect of N and γ o on the performance.…”

“…for illustrating the joint effect of N and γ o on the performance. Readers are referred to Ref [31]. for the characteristics…”

Machine learning and intelligence science: IScIDE (C)

“…[1] and a previous survey in Ref. [9], to the papers [10][11][12][13] in V6N1 and the papers [14][15][16] in V6N2. In this V7N1, the paper by Gao and Wang not only supplements with a brief overview on dimensionality reduction algorithms but also presents the contributions by the team of Prof. Xinbo Gao from Xidian University, another recipient of Chinese NSF Distinguished Young Scholars in 2011.…”