Performance assessment of a cross-validation sampling strategy with active surrogate model selection

Garbo, Andrea; German, Brian

doi:10.1007/s00158-018-02190-7

Cited by 30 publications

(12 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the training phase, machine learning algorithms commonly use cross‐validation techniques to gain higher accuracy. The use of cross‐validation during the training phase is computationally expensive (Garbo & German, 2019; Kang, Qin, Lee, & Lee, 2019). To speed up the training process, 20% of data from each batch is allocated as test instances.…”

Section: Methodsmentioning

confidence: 99%

A causal model for short‐term time series analysis to predict incoming Medicare workload

Mizan

Taghipour

2020

Journal of Forecasting

View full text Add to dashboard Cite

We have investigated methodologies for predicting radiologists' workload in a short time interval by adopting a machine learning technique. Predicting for shorter intervals requires lower execution time combined with higher accuracy. To deal with this issue, an ensemble model is proposed with the fixedbatch-training method. To excel in the execution time, a fixed-batch-training method is used. On the other hand, the ensemble of multiple machine learning algorithms provides higher accuracy. The experimental result shows that this predictive model can produce at least 10% higher accuracy in comparison with the other available widely used short-term time series forecasting models. In the studied medical system, this gain in accuracy for the earlier prediction of workload can reduce the Medicare relative value unit cost by $1.1 million annually, which we have formulated and shown in this paper.The proposed batch-trained ensemble of experts model has also provided at least a 6% improvement in execution time compared with the other studied models.

show abstract

Section: Methodsmentioning

confidence: 99%

A causal model for short‐term time series analysis to predict incoming Medicare workload

Mizan

Taghipour

2020

Journal of Forecasting

View full text Add to dashboard Cite

show abstract

“…409). Shu et al (2018) also uses LOO-CV to select the next point to be simulated; that LOO-CV measures the metamodel accuracy through either the root mean squared prediction error (RMSPE) de…ned in (11) below or the maximum absolute error (MAE) instead of our measure de…ned in (10); Garbo and German (2019) also uses LOO-CV with the MSPE (the square of the RMSPE) to select the next point. Rasmussen and Williams (2006) discusses LOO-CV within a Bayesian framework.…”

Section: Literature Reviewmentioning

confidence: 99%

Statistical Tests for Cross-Validation of Kriging Models

Kleijnen

Beers

2022

INFORMS Journal on Computing

View full text Add to dashboard Cite

Kriging or Gaussian process models are popular metamodels (surrogate models or emulators) of simulation models; these metamodels give predictors for input combinations that are not simulated. To validate these metamodels for computationally expensive simulation models, the analysts often apply computationally efficient cross-validation. In this paper, we derive new statistical tests for so-called leave-one-out cross-validation. Graphically, we present these tests as scatterplots augmented with confidence intervals that use the estimated variances of the Kriging predictors. To estimate the true variances of these predictors, we might use bootstrapping. Like other statistical tests, our tests—with or without bootstrapping—have type I and type II error probabilities; to estimate these probabilities, we use Monte Carlo experiments. We also use such experiments to investigate statistical convergence. To illustrate the application of our tests, we use (i) an example with two inputs and (ii) the popular borehole example with eight inputs. Summary of Contribution: Simulation models are very popular in operations research (OR) and are also known as computer simulations or computer experiments. A popular topic is design and analysis of computer experiments. This paper focuses on Kriging methods and cross-validation methods applied to simulation models; these methods and models are often applied in OR. More specifically, the paper provides the following; (1) the basic variant of a new statistical test for leave-one–out cross-validation; (2) a bootstrap method for the estimation of the true variance of the Kriging predictor; and (3) Monte Carlo experiments for the evaluation of the consistency of the Kriging predictor, the convergence of the Studentized prediction error to the standard normal variable, and the convergence of the expected experimentwise type I error rate to the prespecified nominal value. The new statistical test is illustrated through examples, including the popular borehole model.

show abstract

“…Five individual surrogate models, namely, the KRG (gauss), KRG (exp), RBF (tps), RBF (mq) and PRS (degree2) models, were employed as the initial candidate models (Garbo and German, 2019) (Wang et al, 2014a), the size of the population NP in the PSO model was set to 20D, where D is the dimension of the problem. The maximum allowed number of generations was set to 100 in Problems 1-4, and that in the other problems was set to 300.…”

Section: Effectiveness Of the Proposed Algorithmmentioning

confidence: 99%

Section: Numerical Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

A dynamic adaptive hybrid surrogate-assisted particle swarm optimization algorithm for complex system design optimization

You

Zhang

Niu

2022

View full text Add to dashboard Cite

PurposeSurrogate-assisted evolutionary algorithms (SAEAs) are the most popular algorithms used to solve design optimization problems of expensive and complex engineering systems. However, it is difficult for fixed surrogate models to maintain their accuracy and efficiency in the face of different issues. Therefore, the selection of an appropriate surrogate model remains a significant challenge. This paper aims to propose a dynamic adaptive hybrid surrogate-assisted particle swarm optimization algorithm (AHSM-PSO) to address this issue.Design/methodology/approachA dynamic adaptive hybrid selection method (AHSM) is proposed. This method can identify multiple ensemble models formed by integrating different numbers of excellent individual surrogate models. Then, according to the minimum root-mean-square error, the best suitable surrogate model is dynamically selected in each generation and is used to assist PSO.FindingsExperimental studies on commonly used benchmark problems, and two real-world design optimization problems demonstrate that, compared with existing algorithms, the proposed algorithm achieves better performance.Originality/valueThe main contribution of this work is the proposal of a dynamic adaptive hybrid selection method (AHSM). This method uses the advantages of different surrogate models and eliminates the shortcomings of experience selection. Furthermore, the empirical results of the comparison of the proposed algorithm (AHSM-PSO) with existing algorithms on commonly used benchmark problems, and two real-world design optimization problems demonstrate its competitiveness.

show abstract

Performance assessment of a cross-validation sampling strategy with active surrogate model selection

Cited by 30 publications

References 34 publications

A causal model for short‐term time series analysis to predict incoming Medicare workload

A causal model for short‐term time series analysis to predict incoming Medicare workload

Statistical Tests for Cross-Validation of Kriging Models

A dynamic adaptive hybrid surrogate-assisted particle swarm optimization algorithm for complex system design optimization

Contact Info

Product

Resources

About