Robust generalized cross-validation for choosing the regularization parameter

Abstract

Lukas, M.A. (2006) Robust generalized cross-validation forwhere L i are linear functionals. A prominent method for the selection of the crucial regularization parameter λ is generalized cross-validation (GCV). It is known that GCV has good asymptotic properties as n → ∞ but it may not be reliable for small or medium sized n, sometimes giving an estimate that is far too small. We propose a new robust GCV method (RGCV) which chooses λ to be the minimizer of γV (λ)where V (λ) is the GCV function, F (λ) is an approximate average measure of the influence of each data point on f λ , and γ ∈ (0, 1) is a robustness parameter. We show that for any n, RGCV is less likely than GCV to choose a very small value of λ, resulting in a more robust method.We also show that RGCV has good asymptotic properties as n → ∞ for general linear operator equations with uncorrelated errors. The function EF (λ) approximates the risk ER(λ) for values of λ that are asymptotically a bit smaller than the minimizer of ER(λ) (where V (λ) may not approximate well). The "expected" RGCV estimate is asymptotically optimal as n → ∞ with respect to the "robust risk" γER(λ) where v(λ) is the variance component of the risk, and it has the optimal decay rate with respect to ER(λ) and stronger error criteria. The GCV and RGCV methods are compared in numerical simulations for the problem of estimating the second derivative from noisy data. The results for RGCV with n = 51 are consistent with the asymptotic results, and, for a large range of γ values, RGCV is more reliable and accurate than GCV.

Old and new parameter choice rules for discrete ill-posed problems

“…For many problems, the function G is quite flat around the minimum. The GCV method is known to be unreliable for small to medium-sized problems and robust versions have been developed; see [3,32]. The latter methods require the choice of an additional parameter and, therefore, will not be considered in the present paper.…”

Comparingparameter choice methods for regularization of ill-posed problems

“…However, it is known [37,90,89,102,104,132,142] that for smaller data sets or correlated errors of red noise type, the method is rather unstable, often resulting in under-smoothing. Graphically, the GCV function in (32) can be very flat near its minimum, it can have multiple local A c c e p t e d M a n u s c r i p t minima and the global minimum can be at the extreme endpoint for under-smoothing.…”

Differentiation of matrix functionals using triangular factorization

“…As discussed in the Introduction, in the fitting of splines to noisy data, some criteria for selecting the smoothing parameter require the computation of a functional of the smoothing matrix S(x), where x > 0 is the smoothing parameter. In particular, the GCV criterion requires the computation of tr(S(x)) [11,23], which is the degrees of freedom for the spline, while the RGCV criterion also requires the computation of tr(S 2 (x)) [14], which is the residual degrees of freedom [2] (cf. Section 3.5 in [10]).…”

“…The original motivation for this study was to find an efficient algorithm for calculating the generalized cross-validation (GCV) and robust generalized cross-validation (RGCV) scores for smoothing splines. For GCV [11,23], it is necessary to calculate the trace of the smoothing matrix, S(x) say, while, for RGCV [14], it is also necessary to calculate the trace of S 2 (x). In this situation, the vector of parameters x becomes the scalar smoothing parameter x > 0.…”

“…In this situation, the vector of parameters x becomes the scalar smoothing parameter x > 0. For polynomial smoothing splines of degree 2m − 1, with n + m points, S(x) has the form [14] (1.1) S(x) = I − xCA…”