ENTMOOT: A framework for optimization over ensemble tree models

Abstract

Gradient boosted trees and other regression tree models perform well in a wide range of realworld, industrial applications. These tree models (i) offer insight into important prediction features, (ii) effectively manage sparse data, and (iii) have excellent prediction capabilities. Despite their advantages, they are generally unpopular for decision-making tasks and black-box optimization, which is due to their difficult-to-optimize structure and the lack of a reliable uncertainty measure. ENTMOOT is our new framework for integrating (already trained) tree models into larger optimization problems. The contributions of ENTMOOT include: (i) explicitly introducing a reliable uncertainty measure that is compatible with tree models, (ii) solving the larger optimization problems that incorporate these uncertainty aware tree models, (iii) proving that the solutions are globally optimal, i.e. no better solution exists. In particular, we show how the ENTMOOT approach allows a simple integration of tree models into decision-making and black-box optimization, where it proves as a strong competitor to commonly-used frameworks.Keywords Gradient boosted trees • branch-and-bound • mixed-integer programming • decision-making under uncertainty • black-box optimization

Optimization over decision trees: a case study for the design of stable direct-current electricity networks

“…A recent overview of methods to identify the feasible regions in optimization problems via trained machine learning classifiers can be found in Maragno et al (2021), who present it as a more general framework for data-driven optimization. The works (Mistry et al 2021;Thebelt et al 2021Thebelt et al , 2020Ceccon et al 2022;Thebelt et al 2022) deal with the integration of trained gradient-boosted regression trees into optimization models stemming from different applications. There the distribution of the training data, that influences the prediction accuracy is integrated as a penalty term into the objective function to minimise risk.…”

Machine Learning in Chemical Engineering: A Perspective

“…In the MILP formulations, binary variables are introduced to divide the domain of the piecewise linear ReLU activation functions into two linear sub‐domains. Similarly, tree models can be reformulated as MILPs 58, 64, 65. However, the number of integer variables and constraints grows linearly with the model complexity (e.g., number of nodes in the ANN).…”

Simulation-based optimization of distillation processes using an extended cutting plane algorithm

“…Bayesian optimization is another popular approach to optimize expensive black box functions ( Brochu et al, 2010 ). A black-box optimization framework that combines gradient boosted trees and uncertainty measures to efficiently optimize expensive black box functions was recently presented in ( Thebelt et al, 2020 ). For more details on DFO optimization, we refer to the review papers by ( Boukouvala et al, 2016 ;Rios and Sahinidis, 2012 )…”