We introduce a fast stepwise regression method, called the orthogonal greedy algorithm (OGA), that selects input variables to enter a p-dimensional linear regression model (with p n, the sample size) sequentially so that the selected variable at each step minimizes the residual sum squares. We derive the convergence rate of OGA and develop a consistent model selection procedure along the OGA path that can adjust for potential spuriousness of the greedily chosen regressors among a large number of candidate variables. The resultant regression estimate is shown to have the oracle property of being equivalent to least squares regression on an asymptotically minimal set of relevant regressors under a strong sparsity condition.

The predictive capability of a modification of Rissanen's accumulated prediction error (APE) criterion, APE δn , is investigated in infinite-order autoregressive (AR(∞)) models. Instead of accumulating squares of sequential prediction errors from the beginning, APE δn is obtained by summing these squared errors from stage nδn, where n is the sample size and 1/n ≤ δn ≤ 1 − (1/n) may depend on n. Under certain regularity conditions, an asymptotic expression is derived for the mean-squared prediction error (MSPE) of an AR predictor with order determined by APE δn . This expression shows that the prediction performance of APE δn can vary dramatically depending on the choice of δn. Another interesting finding is that when δn approaches 1 at a certain rate, APE δn can achieve asymptotic efficiency in most practical situations. An asymptotic equivalence between APE δn and an information criterion with a suitable penalty term is also established from the MSPE point of view. This offers new perspectives for understanding the information and prediction-based model selection criteria. Finally, we provide the first asymptotic efficiency result for the case when the underlying AR(∞) model is allowed to degenerate to a finite autoregression.

We consider the problem of choosing the optimal (in the sense of mean-squared prediction error) multistep predictor for an autoregressive (AR) process of finite but unknown order. If a working AR model (which is possibly misspecified) is adopted for multistep predictions, then two competing types of multistep predictors (i.e., plug-in and direct predictors) can be obtained from this model. We provide some interesting examples to show that when both plug-in and direct predictors are considered, the optimal multistep prediction results cannot be guaranteed by correctly identifying the underlying model's order. This finding challenges the traditional model (order) selection criteria, which usually aim to choose the order of the true model. A new prediction selection criterion, which attempts to seek the best combination of the prediction order and the prediction method, is proposed to rectify this difficulty. When the underlying model is stationary, the validity of the proposed criterion is justified theoretically. To obtain this result, asymptotic properties of accumulated squares of multistep prediction errors are investigated. In addition to overcoming the above difficulty, some other advantages of the proposed criterion are also mentioned.

In this paper, a uniform (over some parameter space) moment bound for the inverse of Fisher's information matrix is established. This result is then applied to develop moment bounds for the normalized least squares estimate in (nonlinear) stochastic regression models. The usefulness of these results is illustrated using time series models. In particular, an asymptotic expression for the mean squared prediction error of the least squares predictor in autoregressive moving average models is obtained. This asymptotic expression provides a solid theoretical foundation for some model selection criteria.1. Introduction. Moment inequalities and moment bounds have long been vibrant topics in modern probability and statistics. The celebrated inequalities of Burkholder [3] and Doob [5] offer exemplary illustrations of the importance of moment inequalities. Using moment bounds, the order of magnitude of the spectral norm of the inverse of the Fisher's information matrix can be quantified and consistency and efficiency of least squares estimates of stochastic regression and adaptive control can be established; see, for example, the seminal work of Lai and Wei [15] and the succinct review of Lai and Ying [16]. In this paper, a uniform (over some parameter space) moment bound for the inverse of the Fisher's information matrix is established. This bound is used to investigate the moment properties of least squares estimates and the mean squared prediction error (MSPE) for time series models.

Let observations y 1 , • • • , y n be generated from a first-order autoregressive (AR) model with positive errors. In both the stationary and unit root cases, we derive moment bounds and limiting distributions of an extreme value estimator,ρ n , of the AR coefficient. These results enable us to provide asymptotic expressions for the mean squared error (MSE) ofρ n and the mean squared prediction error (MSPE) of the corresponding predictor,ŷ n+1 , of y n+1 . Based on these expressions, we compare the relative performance ofŷ n+1 (ρ n ) and the least squares predictor (estimator) from the MSPE (MSE) point of view. Our comparison reveals that the better predictor (estimator) is determined not only by whether a unit root exists, but also by the behavior of the underlying error distribution near the origin, and hence is difficult to identify in practice. To circumvent this difficulty, we suggest choosing the predictor (estimator) with the smaller accumulated prediction error and show that the predictor (estimator) chosen in this way is asymptotically equivalent to the better one. Both real and simulated data sets are used to illustrate the proposed method.