Reinforcement learning applied to production planning and control

Abstract

The objective of this paper is to examine the use and applications of reinforcement learning (RL) techniques in the production planning and control (PPC) field addressing the following PPC areas: facility resource planning, capacity planning, purchase and supply management, production scheduling and inventory management. The main RL characteristics, such as method, context, states, actions, reward and highlights, were analysed. The considered number of agents, applications and RL software tools, specifically, programming language, platforms, application programming interfaces and RL frameworks, among others, were identified, and 181 articles were sreviewed. The results showed that RL was applied mainly to production scheduling problems, followed by purchase and supply management. The most revised RL algorithms were model-free and single-agent and were applied to simplified PPC environments. Nevertheless, their results seem to be promising compared to traditional mathematical programming and heuristics/metaheuristics solution methods, and even more so when they incorporate uncertainty or non-linear properties. Finally, RL value-based approaches are the most widely used, specifically Q-learning and its variants and for deep RL, deep Q-networks. In recent years however, the most widely used approach has been the actor-critic method, such as the advantage actor critic, proximal policy optimisation, deep deterministic policy gradient and trust region policy optimisation.

Advanced automatic pass schedule design for hot rolling by coupling reinforcement learning with a fast rolling model

“…The authors analyzed 264 different publications between 2013 and October 2022 and found, that optimizing energy consumption as well as costs and reducing reliance on expert knowledge as the main objectives of these applications. Esteso et al [15] came to similar conclusions and add that the great majority of RL applications in production technology utilize simplified (virtual) environments with discrete action spaces. Panzer and Bender [16] also conducted a literature review and conclude that in numerous applications RL outperforms previously used heuristics or algorithms.…”

A Review on Reinforcement Learning in Production Scheduling: An Inferential Perspective

“…These algorithms, ranked from most to least frequently employed, include Qlearning, temporal difference TD(λ) algorithm, SARSA, ARL, informed Q-learning, dual Q-learning, approximate Q-learning, gradient descent TD(λ) algorithm, revenue sharing, Q-III learning, relational RL, relaxed SMART, and TD(λ)-learning. In the field of DRL, many value-based approaches have been employed, such as DQN (Deep Q-Learning Networks), loosely-coupled DRL, multiclass DQN, and the Q-network algorithm [48][49][50][51][52][53][54][55][56][57][58].…”

Optimizing the microstructure in open-die forgings using reinforcement learning

“…The machine learning approach RL is similar to human trial-and-error learning [3]. In an industrial open-die forging plant, this learning procedure runs unconsciously.…”

“…In this publication, a novel approach for the design of optimized open-die forging processes using the machine learning method reinforcement learning (RL) is presented. RL algorithms as well as deep reinforcement learning (DRL) approaches have been successfully applied in various manufacturing applications including metal forming for different process design and control tasks [3,4]. Gamal et al [5] trained a deep deterministic policy gradient algorithm to control the roll gap in bar and wire hot rolling using real plant measurement data.…”