Q-Learnheuristics: Towards Data-Driven Balanced Metaheuristics

Abstract

One of the central issues that must be resolved for a metaheuristic optimization process to work well is the dilemma of the balance between exploration and exploitation. The metaheuristics (MH) that achieved this balance can be called balanced MH, where a Q-Learning (QL) integration framework was proposed for the selection of metaheuristic operators conducive to this balance, particularly the selection of binarization schemes when a continuous metaheuristic solves binary combinatorial problems. In this work the use of this framework is extended to other recent metaheuristics, demonstrating that the integration of QL in the selection of operators improves the exploration-exploitation balance. Specifically, the Whale Optimization Algorithm and the Sine-Cosine Algorithm are tested by solving the Set Covering Problem, showing statistical improvements in this balance and in the quality of the solutions.

Binarization of Metaheuristics: Is the Transfer Function Really Important?

“…While some MHs operate on binary domains without a binary scheme, studies have demonstrated that continuous MHs supported by a binary scheme perform exceptionally well on multiple NP-hard combinatorial problems [ 1 ]. Examples of such MHs include the binary bat algorithm [ 28 , 29 ], particle swarm optimization [ 30 ], binary sine cosine algorithm [ 2 , 31 , 32 , 33 ], binary salp swarm algorithm [ 34 , 35 ], binary grey wolf optimizer [ 32 , 36 , 37 ], binary dragonfly algorithm [ 38 , 39 ], the binary whale optimization algorithm [ 2 , 32 , 40 ], and the binary magnetic optimization algorithm [ 41 ].…”

“…In the literature, there are several related works on binarization [ 30 , 42 ] that have laid the groundwork for investigations into this domain problem, as there are several practical applications where working in binary domains is necessary. Moreover, research has emerged on how the change of binarization schemes affects each iteration of the search process, such as time-varying binarization schemes [ 38 ] or binarization scheme selectors [ 2 , 32 , 60 ], where the influence of binarization schemes and their impact at both the problem level and each iteration of the search has been demonstrated.…”

“…In the literature, various related works have proposed the hybridization of the sine cosine algorithm, grey wolf optimizer, whale optimization algorithm, and Q-learning [ 2 , 31 , 36 , 40 ]. Q-learning was used as a dynamic binarization scheme selector in each of the metaheuristics, allowing them to solve binary combinatorial problems.…”

“…The BSS provides a way to dynamically choose the binarization scheme, in this case the combination of transfer functions and binarization rules, within a set of them, using the scheme proposed in [ 2 ], where an intelligent operator chooses between a set of actions (possible binarization schemes) by observing the environment (exploration or exploitation).…”

“…As can be seen in Section 2.3 , metaheuristics perform the search process alternating between intensification (exploration) and diversification (exploitation). Previous proposals [ 2 , 31 , 40 , 60 , 61 ] used these phases as the states to use in Q-learning. In these papers, the authors determined the stage of the search process by calculating the population diversity.…”