Asymptotic optimality of generalized cross-validation for choosing the regularization parameter

Abstract

Summary. Let f,x be the regularized solution of a general, linear operator equation, Kfo = g, from discrete, noisy data Yi = g(xi) + ei, i = 1 ..... n, where el are uncorrelated random errors. We consider the prominent method of generalized cross-validation (GCV) for choosing the crucial regularization parameter 2. The practical GCV estimate )~v and its "expected" counterpart 2v are defined as the minimizers of the GCV function V(2) and EV(2), respectively, where E denotes expectation. We investigate the asymptotic performance of 2v with respect to each of the following loss functions: the risk, an L 2 norm on the output error Kf, a --g, and a whole class of stronger norms on the input error f,~ -f0. In the special cases of data smoothing and Fourier differentiation, it is known that as n --. oo, 2v is asymptotically optimal (ao) with respect to the risk criterion. We show this to be true in general, and also extend it to the L 2 norm criterion. The asymptotic optimality is independent of the error variance, the ill-posedness of the problem and the smoothness index of the solution f0-For the input error criterion, it is shown that 2v is weakly ao for a certain class offo if the smoothness offo relative to the regularization space is not too high, but otherwise 2v is sub-optimal. This result is illustrated in the case of numerical differentiation.Mathematics Subject Classifications (1991): 65J10, 62G05, 47A50

Multiframe demosaicing and super-resolution of color images

“…The regularization parameter, λ, is a scalar for properly weighting the first term (data fidelity cost) against the second term (regularization cost). Generally speaking, choosing λ could be either done manually, using visual inspection, or automatically using methods like Generalized Cross-Validation [28], [29], L-curve [30], or other techniques. How to choose such regularization parameters is in itself a vast topic, which we will not treat in the present paper.…”

Comparingparameter choice methods for regularization of ill-posed problems

“…For Tikhonov regularization, it is known [57,95,98] that, with uncorrelated errors, GCV is asymptotically optimal with respect to the prediction risk as the number of data points m → ∞, i.e. the inefficiency goes to 1.…”

“…the inefficiency goes to 1. In addition, if the unknown solution x is not too smooth relative to the operator, then GCV is order optimal for the X -norm risk E x δ n − x 2 [98,139,143]. The condition required for this here is ν ≤ µ + 1/2, and otherwise GCV is order sub-optimal.…”

“…The condition required for this here is ν ≤ µ + 1/2, and otherwise GCV is order sub-optimal. This saturation effect is not as serious as it may seem because (using an approximation of ν) one can choose the order of regularization (order of the Sobolev space X ) so that the condition is satisfied [98]. In fact, one can use GCV to choose the order of regularization [142].…”

Adaptive nonparametric instrumental variables estimation: empirical choice of the regularization parameter

“…Two important characteristics of (1.1)-(1.2) are that (1) the operator A is unknown and must be estimated from the data and (2) the distribution A variety of ways to choose regularization parameters are known in mathematics and numerical analysis. Engl, Hanke, and Neubauer (1996), Mathé and Pereverzev (2003), Bauer and Hohage (2005), Wahba (1977), and Lukas (1993Lukas ( , 1998 describe many. Most of these methods assume that A is known and that the "data" are deterministic or that ( | ) Var Y X x = is known and independent of x .…”