Efficient Hold-Out for Subset of Regressors

Abstract

Abstract. Hold-out and cross-validation are among the most useful methods for model selection and performance assessment of machine learning algorithms. In this paper, we present a computationally efficient algorithm for calculating the hold-out performance for sparse regularized least-squares (RLS) in case the method is already trained with the whole training set. The computational complexity of performing the hold-out is O(|H| 3 + |H| 2 n), where |H| is the size of the hold-out set and n is the number of basis vectors. The algorithm can thus be used to calculate various types of cross-validation estimates effectively. For example, when m is the number of training examples, the complexities of N -fold and leave-one-out cross-validations are O(m 3 /N 2 + (m 2 n)/N ) and O(mn), respectively. Further, since sparse RLS can be trained in O(mn 2 ) time for several regularization parameter values in parallel, the fast hold-out algorithm enables efficient selection of the optimal parameter value.

Efficient cross-validation for kernelized least-squares regression with sparse basis expansions

“…The computational complexity of computing the CV performance with our algorithm is no larger than the complexity of training sparse RLS. The algorithm presented in this section improves our previously proposed one (Pahikkala et al 2009b). Recall that I = {1, .…”

On Learning and Cross-Validation with Decomposed Nyström Approximation of Kernel Matrix

“…However, their method did not remove basis vectors belonging to the holdout set, an approach which we will show to be problematic. A version of this cross-validation algorithm which takes care of this issue for the reduced set approximation of RLS has been introduced by Pahikkala et al [14]. In this work, we generalize the result to arbitrary loss functions with quadratic regularization.…”