Methodology for evaluating optimization experiments

Abstract

The paper presents a methodology for evaluating optimization experiments performed on simulation models (discrete event simulation models and Bin Packing Problem) which focus on various problems in industrial engineering. We created an evaluation application to validate our methodology. The application evaluates the simulation experiments generated by a simulation optimizer. The first goal is to evaluate the behaviour of the tested optimization methods -evaluate a series of optimization experiments (replicated simulation optimization experiments) with different settings of the optimization algorithms according to different characteristics of the optimization (e.g. efficiency of finding a global optimum, quality of a found solution, speed of finding an optimum, etc.). The next goal is to evaluate the difficulty of the simulation optimization -to determine relative frequencies of the number of simulation experiments until the optimum of the objective function is found for different intervals. The objective function represents the quality of a possible solution in the n-dimensional search space for the tested simulation model.

Data Reduction of Digital Twin Simulation Experiments Using Different Optimisation Methods

“…We proposed different evaluation criteria [33]. Each criterion value is between 0 and 1, and it is calculated from the box plot characteristics for the smallest observation, namely, sample minimum Q1, lower quartile Q2, median Q3, and upper quartile Q4, and for the largest observation, sample maximum Q5.…”

“…using discrete-event simulation models. We were also interested in the evaluation of optimisation experiments with different settings of the optimisation methods (big volume of data) from different perspectives [33]. We proposed a methodology that can be applied to the optimisation of several industrial engineering problems represented by a discrete-event simulation model.…”