An Optimum Well Control Using Reinforcement Learning and Policy Transfer; Application to Production Optimization and Slugging Minimization

Abstract

Production optimization of oil, gas and geothermal wells suffering from unstable multiphase flow phenomena such as slugging is a challenging task due to their complexity and unpredictable dynamics. In this work, reinforcement learning which is a novel machine learning based control method was applied to find optimum well control strategies to maximize cumulative production while minimizing the negative impact of slugging on the system integrity, allowing for economical, safe, and reliable operation of wells and flowlines. Actor-critic reinforcement learning agents were trained to find the optimal settings for production valve opening and gas lift pressure in order to minimize slugging and maximize oil production. These agents were trained on a data-driven proxy models of two oil wells with different responses to the control actions. Use of such proxy models allowed for faster modelling of the environment while still accurately representing the system’s physical relations. In addition, to further increase the speed of optimization convergence, a policy transfer schem was developed in which a pre-trained agent on a different well was applied and finetuned on a new well. The reinforcement learning agents successfully managed to learn control strategies that improved oil production by up to 17% and reduced slugging effects by 6% when compared to baseline control settings. In addition, using policy transfer, agents converged up to 63% faster than when trained from a random initialization.