Combining Reinforcement Learning Algorithms with Graph Neural Networks to Solve Dynamic Job Shop Scheduling Problems

Abstract

Smart factories have attracted a lot of attention from scholars for intelligent scheduling problems due to the complexity and dynamics of their production processes. The dynamic job shop scheduling problem (DJSP), as one of the intelligent scheduling problems, aims to make an optimized scheduling decision sequence based on the real-time dynamic job shop environment. The traditional reinforcement learning (RL) method converts the scheduling problem with a Markov process and combines its own reward method to obtain scheduling sequences in different real-time shop states. However, the definition of shop states often relies on the scheduling experience of the model constructor, which undoubtedly affects the optimization capability of the reinforcement learning model. In this paper, we combine graph neural network (GNN) and deep reinforcement learning (DRL) algorithm to solve DJSP. An agent model from job shop state analysis graph to scheduling rules is constructed, thus avoiding the problem that traditional reinforcement learning methods rely on scheduling experience to artificially set the state feature vectors. In addition, a new reward function is defined, and the experimental results prove that our proposed reward method is more effective. The effectiveness and feasibility of our model is demonstrated by comparing with general deep reinforcement learning algorithms on minimizing the earlier and later completion time, which also lays the foundation for solving the DJSP later.

A State Transition Diagram and an Artificial Physarum polycephalum Colony Algorithm for the Flexible Job Shop Scheduling Problem with Transportation Constraints

“…Many research works often independently study job shop scheduling problems and vehicle transportation problems, such as dynamic job shop scheduling [12,17], interval job shop [13], energy-efficient distributed flexible job shop scheduling [14], limited waiting time constraint on a hybrid flowshop [18], embedded environment [19], and flexible job shop scheduling [20]. Ref.…”

“…It is very important and challenging to design efficient algorithms to address it in large-sized cases, such as simulated annealing (SA) [6] and fuzzy logic (FL) [11]. Among them, swarm intelligence (SI) algorithms have received great attention [23,26], i.e., genetic algorithms (GAs) [2,4,16], particle swarm optimiza-tion (PSO) [6,19], ant colony optimization (ACO) [8], deep learning (DL), artificial neural networks (ANNs) [12,27], artificial bee colony (ABC) [13], adaptive memetic algorithms (AMAs) [14], migrating birds optimization [17], grey wolf optimization (GWO) [20], quantum cat swarm optimization [22], artificial slime mold [28], artificial Physarum swarm [29], coronavirus herd immunity [30], artificial plant community [31,32], whale optimization [33], artificial algae [34], and the Jaya algorithm [35]. However, these swarm intelligence algorithms are also prone to fall into local optimization prematurely, and some scholars have tried to improve algorithm performance using hybrid algorithms [6,36].…”

“…Constraint 11 in Equation (12) indicates that on a vehicle v, different jobs are assigned in a first-come-first-served way to ensure that the jobs that arrive first can be transported immediately.…”

“…The latter is also called a transportation scheduling problem or a load/unload problem. Many researchers have proposed all kinds of artificial intelligence algorithms to solve these kinds of complex problems, i.e., genetic algorithms (GAs) [2,4], particle swarm optimization (PSO) [6], simulated annealing (SA) [6], ant colony optimization (ACO) [8], fuzzy logic (FL) [11], deep learning (DL), artificial neural networks (ANNs) [12], artificial bee colony (ABC) [13], and adaptive memetic algorithms (AMAs) [14]. The experimental results of these studies revealed that these artificial intelligence algorithms can obtain satisfactory results in job shop scheduling problems.…”

Fusion Q-Learning Algorithm for Open Shop Scheduling Problem with AGVs

“…However, currently, reinforcement learning is primarily applied to job-shop scheduling problems, with a limited application to open-shop scheduling problems that consider AGV transportation times. Yang et al [35] integrated deep reinforcement learning and graph neural networks to construct an agent model that translates the state of job-shop scheduling problems into scheduling rules using a disjunctive graph representation. They applied this approach to solve job-shop problems with the objective of minimizing makespan, demonstrating the feasibility of the method.…”