Decision-based model selection

Boer, Alexander; Sierag, Dirk

doi:10.1016/j.ejor.2020.08.025

Cited by 7 publications

(4 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results showing that data specific model selection yields an improvement over random model selection may be explained by considering the differing strengths of these algorithms. Some of these are better at modeling non-linear effects, while others are better at extrapolating and interpolating data [27]. It is then beneficial to have a systematic method for modeling any data utilizing all the algorithms mentioned in this study and comparing their predictive performance.…”

Section: Discussionmentioning

confidence: 99%

Performance comparison of ten state-of-the-art machine learning algorithms for outcome prediction modeling of radiation-induced toxicity

Salazar,

Nair,

Leone

et al. 2024

Preprint

View full text Add to dashboard Cite

Purpose: To evaluate the efficacy of prominent machine learning algorithms in predicting normal tissue complication probability utilizing clinical data obtained from two distinct disease sites, and to create a software tool that facilitates the automatic determination of the optimal algorithm to model any given labeled dataset. Methods and Materials: We obtained 3 sets of radiation toxicity data (478 patients) from our clinic, gastrointestinal toxicity (GIT), radiation pneumonitis (RP), and radiation esophagitis (RE). These data comprised clinicopathological and dosimetric information for patients diagnosed with non-small cell lung cancer and anal squamous cell carcinoma. Each dataset was modeled using ten commonly employed machine learning algorithms (elastic net, LASSO, random forest, regression forest, support vector machine, XGBoost, k-nearest-neighbors, neural network, Bayesian-LASSO, and Bayesian neural network) by randomly dividing the dataset into a training and test set. The training set was used to create and tune the model, and the test set served to assess it by calculating performance metrics. This process was repeated 100 times by each algorithm for each dataset. Figures were generated to visually compare the performance of the algorithms. A graphical user interface was developed to automate this whole process. Results: LASSO achieved the highest area under the precision-recall curve (AUPRC) (0.807±0.067) for RE, random forest for GIT (0.726±0.096), and the neural network for RP (0.878±0.060). Area-under-the-curve was 0.754±0.069, 0.889±0.043, and 0.905±0.045, respectively. The graphical user interface was used to compare all algorithms for each dataset automatically. When averaging AUPRC across all toxicities, Bayesian-LASSO was the best model. Conclusion: Our results show that there is no best algorithm for all datasets. Therefore, it is important to compare multiple algorithms when training an outcome prediction model on a new dataset. The graphical user interface created for this study automatically compares the performance of these ten algorithms for any dataset.

show abstract

Section: Discussionmentioning

confidence: 99%

Performance comparison of ten state-of-the-art machine learning algorithms for outcome prediction modeling of radiation-induced toxicity

Salazar,

Nair,

Leone

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…A general approach has recently been proposed by den Boer and Sierag (2020). Under the assumption of the continuity of the decision space, they show that their method is consistent and delivers the true optimum decision, i.e.…”

Section: Literature Reviewmentioning

confidence: 99%

Evaluating Predictive Models in the Orienteering Problem with Stochastic Profits: A Simulation Study

Çakır

2021

isarder

View full text Add to dashboard Cite

The aim of this paper is to test the effectiveness of statistical model selection measures in terms of decision quality for the orienteering problem with stochastic profits using simulation.Design/methodology/approach -This paper is based on a quantitative numerical approach where various model selection measures are evaluated using computational experiments including modelbased computer-generated random data.Findings -The findings of this paper include experimental results showing a deficiency of about 6.5 units of classical selection measures relative to a decision-based selection measure for the Tsiligirides orienteering benchmark instances.Discussion -While classical model selection measures are suitable for accuracy reasons, misspecified models sometimes do lead to better decision outcomes. From a practical perspective, in order to carry out prescriptive analytics for orienteering problems, having access to a reasonable decision algorithm at the prediction stage of data-analysis can be beneficial for downstream realized profit.

show abstract

“…This motivates the recent "predict-thenoptimize" approach to model preference and a couple examples. Den Boer et al (2021) briefly review work on statistical criteria and bridging model selection with data-driven optimization, then contribute their own resampling approach to select models maximizing decision reward. They emphasize how misspecified models can often result in better decisions.…”

Section: Introductionmentioning

confidence: 99%

Robust Bellman State Prediction with Learning and Model Preferences

Estey

2024

Preprint

View full text Add to dashboard Cite

I contribute to stochastic modeling methodology in a theoretical framework spanning core decisions in the model's lifetime. These are predicting an out-of-sample unit's latent state even from non-series data, deciding when to start and stop learning about the state variable, and choosing models from important trade-offs. States evolve from linear dynamics with time-varying predictors and coefficients (drift) and generalized continuous noise (diffusion). Coefficients must address misprediction costs, data complexity, and distributional uncertainty (ambiguity) about the state's diffusion and stopping time. I exactly solve a stochastic dynamic program robust to worst-case costs from both uncertainties. The Bellman optimal coefficients extend generalized ridge regression by out-of-sample components impacting value changes given state changes. Performance issues trigger sequential analysis whether learning alternative models, given the effort, is better than keeping baseline. Learning is method-general and stops in fewest average attempts within decision errors. I derive preference functions for comparing models with state and cost-change constraints to decide a model, joint-time state and value distributions, and other properties beneficial to modelers.

show abstract

Decision-based model selection

Cited by 7 publications

References 38 publications

Performance comparison of ten state-of-the-art machine learning algorithms for outcome prediction modeling of radiation-induced toxicity

Performance comparison of ten state-of-the-art machine learning algorithms for outcome prediction modeling of radiation-induced toxicity

Evaluating Predictive Models in the Orienteering Problem with Stochastic Profits: A Simulation Study

Robust Bellman State Prediction with Learning and Model Preferences

Contact Info

Product

Resources

About