Intelligent Inventory Control: Is Bootstrapping Worth Implementing?

Abstract

The common belief is that using Reinforcement Learning methods (RL) with bootstrapping gives better results than without. However, inclusion of bootstrapping increases the complexity of the RL implementation and requires significant effort. This study investigates whether inclusion of bootstrapping is worth the effort when applying RL to inventory problems. Specifically, we investigate bootstrapping of the temporal difference learning method by using eligibility trace. In addition, we develop a new bootstrapping extension to the Residual Gradient method to supplement our investigation. The results show questionable benefit of bootstrapping when applied to inventory problems. Significance tests could not confirm that bootstrapping had statistically significantly reduced costs of inventory controlled by a RL agent. Our empirical results are based on a variety of problem settings, including demand correlations, demand variances, and cost structures.

Intelligent Inventory Control via Ruminative Reinforcement Learning

“…Relying on learning mechanism, RL in its typical form does not require knowledge of a structure of the problem. Therefore, RL has been studied in wide range of sequential decision problems, for example, virtual machine configuration [4], robotics [5], helicopter control [6], ventilation, heating and air conditioning control [7], electricity trade [8], financial management [9], water resource management [10], and inventory management [11]. Acceptance of RL is credited to RL's effectiveness, potential possibilities [12], link to mammal learning processes [13,14], and its model-free property [15].…”

Ruminative Reinforcement Learning: Improve Intelligent Inventory Control by Ruminating on the Past

“…Reinforcement Learning (RL) is an approach to solve practical sequential decision or sophisticated control problems, even when structures of the problems are less understood. RL has been studied extensively and applied in wide range of applications, including virtual machine configuration [1], robotics [2], helicopter control [3], ventilation, heating and air conditioning control [4], financial management [5], water resource management [6], and inventory management [7]. Prevalence of RL in current research is credited to RL's effectiveness, potential possibilities [8], link to mammal learning processes [9], and its model-free property [10].…”