AIC for the non-concave penalized likelihood method

Abstract

Non-concave penalized maximum likelihood methods, such as the Bridge, the SCAD, and the MCP, are widely used because they not only perform the parameter estimation and variable selection simultaneously but also are more efficient than the Lasso. They include a tuning parameter which controls a penalty level, and several information criteria have been developed for selecting it. While these criteria assure the model selection consistency, they have a problem in that there are no appropriate rules for choosing one from the class of information criteria satisfying such a preferred asymptotic property. In this paper, we derive an information criterion based on the original definition of the AIC by considering minimization of the prediction error rather than model selection consistency. Concretely speaking, we derive a function of the score statistic that is asymptotically equivalent to the non-concave penalized maximum likelihood estimator and then provide an estimator of the Kullback-Leibler divergence between the true distribution and the estimated distribution based on the function, whose bias converges in mean to zero.Furthermore, through simulation studies, we find that the performance of the proposed information criterion is about the same as or even better than that of the cross-validation.

Moment convergence in regularized estimation under multiple and mixed-rates asymptotics

“…We may also apply our moment-convergence results to validate the celebrated AIC methodology ( [3], and also [10], [26], and [27]) even under the sparse asymptotics. Recent studies exactly in this direction contain [25] and [29], where the uniform integrability of the sparse maximum-likelihood estimator with the bridge-like regularization played an important role for validating the asymptotic bias correction. Below we will briefly discuss how we can extend the result of [29] to cover a broader range of statistical models with locally asymptotically normal structure.…”

“…Recent studies exactly in this direction contain [25] and [29], where the uniform integrability of the sparse maximum-likelihood estimator with the bridge-like regularization played an important role for validating the asymptotic bias correction. Below we will briefly discuss how we can extend the result of [29] to cover a broader range of statistical models with locally asymptotically normal structure. To keep things simple, we are only concerned here with correctly specified models, keeping the multi-scaling setup with θ = (α, β); again we note that cases of single scaling can be handled as well without any essential change, just by ignoring the β-part.…”

“…In that case, it may even happen that the random function M n (u; θ 0 ) diverges in probability for each u; indeed, the sparse-type estimation, which is a particular case of regularized estimation, falls into the region of mixed-rates asymptotics. In this case, convergence of moments does not follow from a direct application of [33], and we are aware of only the following previous studies in this direction: [29] deduced the convergence of moments of a regularized sparse maximumlikelihood estimator of the generalized linear model, and applied it to verify the AIC type variable selection; [25] (also [20]) deduced the moment convergence in regularized estimation of a linear regression model with general regularization term. Nevertheless, the proofs of these results made particular use of the special structure of the considered models and/or the convexity argument, and both of them do not tell us much about the case of general regularized M -estimation.…”

Oracle inequalities for sign constrained generalized linear models

“…2 An important conclusion of these studies is that sign-constraints themselves can be as effective as more explicit regularization methods such as the Lasso if the design matrix satisfies a special condition called the positive eigenvalue condition (which is called the self-regularizing property in [18]) in addition to a compatibility condition, where the latter condition is a standard one to prove the theoretical properties of regularization methods. More precisely, Meinshausen [14] has derived oracle inequalities for the ℓ 1 -error of the regression 1 To avoid such a computational burden, some authors have developed analytical methods for choosing the tuning parameter λ such as using an information criterion; see [15,20] and references therein for details. 2 We remark that the statistical property of the NNLS estimator in a fixed-dimensional setting has been extensively studied in the literature from a long ago; see e.g.…”

Partial quasi-likelihood analysis

“…The importance of such precise estimates of tail probability is well recognized in asymptotic decision theory, prediction, theory of information criteria for model selection, asymptotic expansion, etc. The QLA is rapidly expanding the range of its applications: for example, sampled ergodic diffusion processes (Yoshida [30]), contrastbased information criterion for diffusion processes (Uchida [23]), approximate self-weighted LAD estimation of discretely observed ergodic Ornstein-Uhlenbeck processes (Masuda [12]), jump diffusion processes Ogihara and Yoshida([17]), adaptive estimation for diffusion processes (Uchida and Yoshida [24]), adaptive Bayes type estimators for ergodic diffusion processes (Uchida and Yoshida [27]), asymptotic properties of the QLA estimators for volatility in regular sampling of finite time horizon (Uchida and Yoshida [25]) and in non-synchronous sampling (Ogihara and Yoshida [18]), Gaussian quasi-likelihood random fields for ergodic Lévy driven SDE (Masuda [15]), hybrid multi-step estimators (Kamatani and Uchida [6]), parametric estimation of Lévy processes (Masuda [13]), ergodic point processes for limit order book (Clinet and Yoshida [1]), a non-ergodic point process regression model (Ogihara and Yoshida [19]), threshold estimation for stochastic processes with small noise (Shimizu [21]), AIC for non-concave penalized likelihood method (Umezu et al [28]), Schwarz type model comparison for LAQ models (Eguchi and Masuda [2]), adaptive Bayes estimators and hybrid estimators for small diffusion processes based on sampled data (Nomura and Uchida [16]), moment convergence of regularized leastsquares estimator for linear regression model (Shimizu [22]), moment convergence in regularized estimation under multiple and mixed-rates asymptotics (Masuda and Shimizu [14]), asymptotic expansion in quasi likelihood analysis for volatility (Yoshida [31]) among others.…”