Hierarchical Reinforcement Learning for Multi-Objective Real-Time Flexible Scheduling in a Smart Shop Floor

Abstract

With the development of intelligent manufacturing, machine tools are considered the “mothership” of the equipment manufacturing industry, and the associated processing workshops are becoming more high-end, flexible, intelligent, and green. As the core of manufacturing management in a smart shop floor, research into the multi-objective dynamic flexible job shop scheduling problem (MODFJSP) focuses on optimizing scheduling decisions in real time according to changes in the production environment. In this paper, hierarchical reinforcement learning (HRL) is proposed to solve the MODFJSP considering random job arrival, with a focus on achieving the two practical goals of minimizing penalties for earliness and tardiness and reducing total machine load. A two-layer hierarchical architecture is proposed, namely the combination of a double deep Q-network (DDQN) and a dueling DDQN (DDDQN), and state features, actions, and external and internal rewards are designed. Meanwhile, a personal computer-based interaction feature is designed to integrate subjective decision information into the real-time optimization of HRL to obtain a satisfactory compromise. In addition, the proposed HRL framework is applied to multi-objective real-time flexible scheduling in a smart gear production workshop, and the experimental results show that the proposed HRL algorithm outperforms other reinforcement learning (RL) algorithms, metaheuristics, and heuristics in terms of solution quality and generalization and has the added benefit of real-time characteristics.

A Hybrid-Model-Based CNC Machining Trajectory Error Prediction and Compensation Method

“…Intelligent manufacturing refers to a new type of manufacturing that can create and deliver products and services through the integrated and intelligent utilization of processes and resources in the information and physical spaces of different system hierarchies [21,22]. With the promotion and application of intelligent manufacturing technology in aerospace and other precision machining fields, intelligent application service demands such as self-awareness, self-decision-making, self-execution, self-learning, and self-optimization have been put made of CNC systems [23].…”

“…The concatenation result of the concat function is the merging of the calculations from each attention head. The definition of the attention function is given by Equation (22).…”

Efficient Multi-Objective Optimization on Dynamic Flexible Job Shop Scheduling Using Deep Reinforcement Learning Approach

“…Abebaw et al (2023) [22] considered the JSSP as an iterative decisionmaking problem, and a DDQN is utilized for training the model and learning an optimal policy in which six continuous state features are formulated to record the production environment; an epsilon-greedy strategy is used on the action selection; furthermore, the reward and the penalty of the evaluation metric are designed. Zhang et al (2022) [23] used the PPO algorithm in the DRL framework to tackle the dynamic scheduling problem in a job shop manufacturing system with an unexpected event of the machine failure in which the transport agent is required to dispatch jobs/orders to machines then to sinks from machines after the task of jobs is completed. The proposed framework was validated based on the real-world job shop manufacturing system.…”

“…For variant production scenarios, it is indispensable to independently propose the corresponding appropriate multi-objective optimization schemes. In the JSP [4][5][6][7], the product processing of each operation can only be appointed one machine, while the addition of the flexible characteristic in the FJSP [8][9][10][11] can make it more conformed to a real-world production circumstance in which all operations of a product can be processed using more than one available machine. The dynamic JSP or FJSP emphasizes the adaptability of the trained model to handle interruptive incidents and still obtain a complete scheduling solution that is approximately optimal.…”

Fusion Q-Learning Algorithm for Open Shop Scheduling Problem with AGVs

“…Simulation experiments ultimately demonstrated that their proposed framework significantly improves solution quality. Chang et al [45] employed a combination of double-layered Q-learning and deep Q-learning to address multi-objective flexible problems, demonstrating superior solution quality and generalization capability in the obtained results. Lin et al [46] devised two Q-learning-based strategies to enhance the solving capability of local search, substantiating the effectiveness of their approach through a case study.…”