Kernelizing PLS, degrees of freedom, and efficient model selection

Krämer, Nicole; Braun, Mikio L.

doi:10.1145/1273496.1273552

Cited by 24 publications

(26 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…negative Degrees of Freedom). In Krämer and Braun (2007), the sparse structure of L is used and an additional stopping criterion is imposed to ensure that the latent components are orthogonal. However, Figure 6: Illustration: Scaled mean test error as a function of the number of components.…”

Section: Numerical Stability and Runtimementioning

confidence: 99%

The Degrees of Freedom of Partial Least Squares Regression

Krämer

Sugiyama

2011

Journal of the American Statistical Association

Self Cite

View full text Add to dashboard Cite

The derivation of statistical properties for Partial Least Squares regression can be a challenging task. The reason is that the construction of latent components from the predictor variables also depends on the response variable. While this typically leads to good performance and interpretable models in practice, it makes the statistical analysis more involved. In this work, we study the intrinsic complexity of Partial Least Squares Regression. Our contribution is an unbiased estimate of its Degrees of Freedom. It is defined as the trace of the first derivative of the fitted values, seen as a function of the response. We establish two equivalent representations that rely on the close connection of Partial Least Squares to matrix decompositions and Krylov subspace techniques. We show that the Degrees of Freedom depend on the collinearity of the predictor variables: The lower the collinearity is, the higher the Degrees of Freedom are. In particular, they are typically higher than the naive approach that defines the Degrees of Freedom as the number of components. Further, we illustrate how our Degrees of Freedom estimate can be used for the comparison of different regression methods. In the experimental section, we show that our Degrees of Freedom estimate in combination with information criteria is useful for model selection.

show abstract

Section: Numerical Stability and Runtimementioning

confidence: 99%

The Degrees of Freedom of Partial Least Squares Regression

Krämer

Sugiyama

2011

Journal of the American Statistical Association

Self Cite

View full text Add to dashboard Cite

show abstract

“…, Me p } was obtained from Eq. (28), and the corresponding set of numbers of latent variables for the gates Mg = {Mg 2 , . .…”

Section: Evaluation and Discussionmentioning

confidence: 99%

“…Usually the DOF is set to be equal to the number of latent variables, but this is a wrong assumption and does not lead to satisfactory results in the selection of the number of latent variables [28,29]. This problem of determining the DOF in a PLS model was addressed in [29], where an unbiased estimate of the DOF has been proposed.…”

Section: Selecting the Number Of Latent Variablesmentioning

confidence: 99%

Mixture of partial least squares experts and application in prediction settings with multiple operating modes

Souza

Araújo

2014

Chemometrics and Intelligent Laboratory Systems

View full text Add to dashboard Cite

This paper addresses the problem of online quality prediction in processes with multiple operating modes. The paper proposes a new method called mixture of partial least squares regression (Mix-PLS), where the solution of the mixture of experts regression is performed using the partial least squares (PLS) algorithm. The PLS is used to tune the model experts and the gate parameters. The solution of Mix-PLS is achieved using the expectation-maximization (EM) algorithm, and at each iteration of EM algorithm the number of latent variables of the PLS for the gate and experts are determined using the Bayesian information criterion. The proposed method, shows to be less prone to overfitting with respect to the number of mixture models, when compared to the standard mixture of linear regression experts (MLRE). The Mix-PLS was successfully applied on three real prediction problems. The results were compared with five other regression algorithms. In all the experiments, the proposed method always exhibits the best prediction performance.

show abstract

“…6.7 in [3]. Although PLS is usually introduced as an iterative procedure without explicit objective functions, it was shown [27] that kernel PLS minimizes the same objective function as ordinary least squares, but the solution is restricted within a subspace spanned by Ky, K 2 y, . .…”

Section: Partial Least Squaresmentioning

confidence: 99%

“…To determine r, see [27] for a comparison among various criteria. In all kernelized PLS algorithms known so far, the computational cost is dominated by the matrix-vector products.…”

Section: Partial Least Squaresmentioning

confidence: 99%

Recent Advances and Trends in Large-Scale Kernel Methods

Kashima

Idé

Kato

et al. 2009

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYKernel methods such as the support vector machine are one of the most successful algorithms in modern machine learning. Their advantage is that linear algorithms are extended to non-linear scenarios in a straightforward way by the use of the kernel trick. However, naive use of kernel methods is computationally expensive since the computational complexity typically scales cubically with respect to the number of training samples. In this article, we review recent advances in the kernel methods, with emphasis on scalability for massive problems.

show abstract

Kernelizing PLS, degrees of freedom, and efficient model selection

Cited by 24 publications

References 11 publications

The Degrees of Freedom of Partial Least Squares Regression

The Degrees of Freedom of Partial Least Squares Regression

Mixture of partial least squares experts and application in prediction settings with multiple operating modes

Recent Advances and Trends in Large-Scale Kernel Methods

Contact Info

Product

Resources

About