Effective reinforcement learning through evolutionary surrogate-assisted prescription

Abstract

There is now significant historical data available on decision making in organizations, consisting of the decision problem, what decisions were made, and how desirable the outcomes were. Using this data, it is possible to learn a surrogate model, and with that model, evolve a decision strategy that optimizes the outcomes. This paper introduces a general such approach, called Evolutionary Surrogate-Assisted Prescription, or ESP. The surrogate is, for example, a random forest or a neural network trained with gradient descent, and the strategy is a neural network that is evolved to maximize the predictions of the surrogate model. ESP is further extended in this paper to sequential decision-making tasks, which makes it possible to evaluate the framework in reinforcement learning (RL) benchmarks. Because the majority of evaluations are done on the surrogate, ESP is more sample efficient, has lower variance, and lower regret than standard RL approaches. Surprisingly, its solutions are also better because both the surrogate and the strategy network regularize the decision making behavior. ESP thus forms a promising foundation to decision optimization in real-world problems.

A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning

“…For instance, Evolutionary Surrogate-assisted Prescription (ESP) [41] incorporates a surrogate model into the EA. Given a set of input states , the policy neural network takes each ∈ as the input and outputs the action .…”

“…In general, apart from limited successes on simple discrete control tasks such as Cart-Pole [19], conventional surrogate-assisted EAs [27,40,41] still face significant challenges. To explore the potential of surrogate-assisted methods in continuous and complex RL contexts, in this work, we extend the surrogate model to hybrid RL frameworks with the objective to reduce the computational cost while making full use of the efficiency of the RL agent and the exploration capability of the EA.…”

“…The results in Figure 3 suggest that, at the beginning of training, the surrogate has relatively low approximation accuracy. However, previous studies [32,41,44] suggest that this kind of uncertainty may not cause negative effects for evolutionary search. Instead, it may further push the population to explore the fitness landscape.…”

“…For sequential decision-making problems, the genotype-phenotype-fitness mapping is often difficult to learn but it is straightforward for the surrogate model to estimate the outcome of a stateaction pair ( , ). In ESP [41], individuals are evaluated by an extra dedicated surrogate model. However, in hybrid RL frameworks, the critic ( , | ) of the RL agent can be naturally used as a surrogate.…”

Creative AI Through Evolutionary Computation: Principles and Examples

“…The decision maker can then select a desired tradeoff, and the Prescriptor will recommend the best NPI strategy that achieves it. This process, Evolutionary Surrogate-assisted Prescription (ESP; [7]), made several creative discoveries. Early on it recognized that schools and workplaces are the most important NPIs; Indeed these are the two activities where people spend a lot of time with other people indoors, where it is possible to be exposed to significant amounts of the virus, as later became evident [15,19,27].…”