Optimizing Objective Functions Determined from Random Forests

Biggs, Max; Hariss, Rim; Perakis, Georgia

doi:10.2139/ssrn.2986630

Cited by 28 publications

(27 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To our knowledge, Biggs et al (2017) is the only work in the PMO literature that has considered a notion of training relevance. The authors study a property investment problem and ensure similarity between solutions found via a PMO model and the historical data by constraining the search space to be within the convex hull of the features of historical observations.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Other application areas include food deliveryLiu et al (2020), scholarship allocationBergman et al (2019), personalized pricingBiggs et al (2021), and auctionsVerwer et al (2017).Optimization problems defined over trained predictive models are often challenging to solve, depending on the properties of the underlying predictive model Mišić (2020). andBiggs et al (2017) propose exact solution approaches for optimization problems defined over tree ensembles.Mixed integer programming (MIP) formulations for optimization models defined over trained neural networks are studied byCheng et al (2017),Fischetti and Jo (2018),Bunel et al (2018), Dutta et al (2018,Bergman et al (2019),Grimstad and Andersson (2019a),Schweidtmann and Mitsos (2019),Tjeng et al (2019),Grimstad and Andersson (2019b),Botoeva et al (2020), Anderson et al (2020 andTsay et al (2021). Ensembles of neural networks are studied byWang et al…”

mentioning

confidence: 99%

See 1 more Smart Citation

Constraint Learning to Define Trust Regions in Predictive-Model Embedded Optimization

Shi¹,

Emadikhiav²,

Lozano³

et al. 2022

Preprint

View full text Add to dashboard Cite

There is a recent proliferation of research on the integration of machine learning and optimization. One expansive area within this research stream is predictive-model embedded optimization, which uses pre-trained predictive models for the objective function of an optimization problem, so that features of the predictive models become decision variables in the optimization problem. Despite a recent surge in publications in this area, one aspect of this decision-making pipeline that has been largely overlooked is training relevance, i.e., ensuring that solutions to the optimization problem should be similar to the data used to train the predictive models. In this paper, we propose constraints designed to enforce training relevance, and show through a collection of experimental results that adding the suggested constraints significantly improves the quality of solutions obtained.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

mentioning

confidence: 99%

Constraint Learning to Define Trust Regions in Predictive-Model Embedded Optimization

Shi¹,

Emadikhiav²,

Lozano³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…As embedding predictive models into optimization formulations is a critical aspect of this framework, previous literature has focused on predictive models that can be linearized and formulated as a mixed-integer linear programs (MILP), such as logistic regression, linear models, decision trees, random forests, and neural networks with Rectified Linear Unit (ReLU) activation functions (Bergman et al 2019, Verwer et al 2017, Biggs et al 2017, Mišić 2020. Some predictive models and their regularized versions lead to easier optimization problems and are chosen to eschew computational intractability (Liu et al 2020, Bertsimas et al 2016.…”

Section: Literature Reviewmentioning

confidence: 99%

Optimizing over an ensemble of neural networks

Wang¹,

Lozano²,

Cardonha³

et al. 2021

Preprint

View full text Add to dashboard Cite

We study optimization problems where the objective function is modeled through feedforward neural networks with rectified linear unit (ReLU) activation. Recent literature has explored the use of a single neural network to model either uncertain or complex elements within an objective function. However, it is well known that ensembles of neural networks produce more stable predictions and have better generalizability than models with single neural networks, which suggests the application of ensembles of neural networks in a decision-making pipeline. We study how to incorporate a neural network ensemble as the objective function of an optimization model and explore computational approaches for the ensuing problem. We present a mixed-integer linear program based on existing popular big-𝑀 formulations for optimizing over a single neural network. We develop two acceleration techniques for our model, the first one is a preprocessing procedure to tighten bounds for critical neurons in the neural network while the second one is a set of valid inequalities based on Benders decomposition. Experimental evaluations of our solution methods are conducted on one global optimization problem and two real-world data sets; the results suggest that our optimization algorithm outperforms the adaption of an state-of-the-art approach in terms of computational time and optimality gaps.

show abstract

“…Current research focus has been on applying machine learning methodologies to predict the counterfactuals, based on which optimal decisions can be made. Local learning methods such as K-Nearest Neighbors (Altman, 1992), LOESS (LOcally Estimated Scatterplot Smoothing) (Cleveland and Devlin, 1988), CART (Classification And Regression Trees) (Breiman, 2017), and Random Forests (Breiman, 2001), have been studied inBertsimas and Kallus, 2019; Bertsimas et al, 2019a;Dunn, 2018;Biggs and Hariss, 2018. Extensions to continuous and multi-dimensional decision spaces with observational data were considered in McCord, 2018. To prevent overfitting, Bertsimas andVan Parys, 2017 proposed two robust prescriptive methods based on Nadaraya-Watson and nearest-neighbors learning.…”

Section: The Problem and Related Workmentioning

confidence: 99%

Distributionally Robust Learning

Chen

Paschalidis

2020

FNT in Optimization

View full text Add to dashboard Cite

This monograph develops a comprehensive statistical learning framework that is robust to (distributional) perturbations in the data using Distributionally Robust Optimization (DRO) under the Wasserstein metric. Beginning with fundamental properties of the Wasserstein metric and the DRO formulation, we explore duality to arrive at tractable formulations and develop finite-sample, as well as asymptotic, performance guarantees. We consider a series of learning problems, including (i) distributionally robust linear regression; (ii) distributionally robust regression with group structure in the predictors; (iii) distributionally robust multi-output regression and multiclass classification, (iv) optimal decision making that combines distributionally robust regression with nearest-neighbor estimation; (v) distributionally robust semi-supervised learning, and (vi) distributionally robust reinforcement learning. A tractable DRO relaxation for each problem is being derived, establishing a connection between robustness and regularization, and obtaining bounds on the prediction and estimation errors of the solution. Beyond theory, we include numerical experiments and case studies using synthetic and real data. The real data experiments are all associated with various health informatics problems, an application area which provided the initial impetus for this work.

show abstract

Optimizing Objective Functions Determined from Random Forests

Cited by 28 publications

References 41 publications

Constraint Learning to Define Trust Regions in Predictive-Model Embedded Optimization

Constraint Learning to Define Trust Regions in Predictive-Model Embedded Optimization

Optimizing over an ensemble of neural networks

Distributionally Robust Learning

Contact Info

Product

Resources

About