Controlling Commercial Cooling Systems Using Reinforcement Learning

Luo, Jie; Păduraru, Cosmin; Voicu, Octavian; Chervonyi, Yuri; Munns, Scott A.; Li, Jerry; Qian, Crystal; Dutta, Praneet; Davis, Jared Quincy; Wu, Ningjia; Yang, Xiaofan; Chang, Chu-Ming; Li, Ted; Rose, Robert J.; Fan, Mingyan; Nakhost, Hootan; Liu, Tinglin; Kirkman, Brian; Altamura, Frank; Cline, Lee; Tonker, Patrick; Gouker, Joel; Uden, Dave; Bryan, Warren Buddy; Law, Jason; Fatiha, Deeni; Satra, Neil; Rothenberg, Juliet; Carlin, Molly; Tallapaka, Satish; Witherspoon, Sims; Parish, David J.; Dolan, Peter; Chen, Zhao; Mankowitz, Daniel J.

doi:10.48550/arxiv.2211.07357

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…En la industria se encuentran aplicaciones que utilizan un esquema de GPI con restricciones y una arquitectura DQN en la optimización de sistemas de refrigeración para centros de datos (Luo et al, 2022). Por otra parte, Netflix utiliza SARSA y Q-Learning en sus nuevos sistemas de recomendación (Elahi, 2022), mientras que IBM los emplea en sus sistemas de predicción de mercado.…”

Section: Aplicaciones Exitosas De Rlunclassified

Una introducciön amable pero rigurosa al aprendizaje por refuerzo

Montenegro Meza,

Méndez,

Méndez

2023

recibe

View full text Add to dashboard Cite

La interacciön con el mundo es una de las principales formas en las que se genera el apren- dizaje, pues es el medio por el cuål se obtiene informaciön del entorno, y se experimentan rela- ciones causa-efecto. Esta idea de aprender mediante la interacciön es una aspecto fundamental en muchas teorias del aprendizaje y, en este articulo abordaremos un enfoque computacional llamado aprendizaje por refuerzo (Reinforcement Learning, RL) ademås de construir de manera progresiva y sencilla sus bases matemåticas, asi como los métodos principales de soluciön. Por ültimo, mostramos aplicaciones y algoritmos que son relevantes en la industria e investigaciån.

show abstract

Section: Aplicaciones Exitosas De Rlunclassified

Una introducciön amable pero rigurosa al aprendizaje por refuerzo

Montenegro Meza,

Méndez,

Méndez

2023

recibe

View full text Add to dashboard Cite

show abstract

“…As we can see in the plot of Feed Turbidity, some days are similar, but others, such as May 31, 2022, exhibit dramatically different dynamics. In some applications like HVAC control (Luo et al, 2022), it is sufficient to perform learning on a batch of data once a day. In water treatment, the sensor dynamics provide the opportunity to observe sensor changes throughout the day.…”

Section: Sensor Drift and Seasonal Changementioning

confidence: 99%

“…Closer to our work, recent work on automating HVAC control used an approach where the agent is first tuned on off-line data and then a learning controller is deployed that is updated once a day (Luo et al, 2022). In this work the authors explicitly avoided offline training on operator data, citing the well-known issues of insufficient action coverage.…”

Section: Introductionmentioning

confidence: 99%

GVFs in the real world: making predictions online for water treatment

Janjua,

Shah,

White

et al. 2023

Mach Learn

View full text Add to dashboard Cite

In this paper we investigate the use of reinforcement-learning based prediction approaches for a real drinking-water treatment plant. Developing such a prediction system is a critical step on the path to optimizing and automating water treatment. Before that, there are many questions to answer about the predictability of the data, suitable neural network architectures, how to overcome partial observability and more. We first describe this dataset, and highlight challenges with seasonality, nonstationarity, partial observability, and heterogeneity across sensors and operation modes of the plant. We then describe General Value Function (GVF) predictions—discounted cumulative sums of observations–and highlight why they might be preferable to classical n-step predictions common in time series prediction. We discuss how to use offline data to appropriately pre-train our temporal difference learning (TD) agents that learn these GVF predictions, including how to select hyperparameters for online fine-tuning in deployment. We find that the TD-prediction agent obtains an overall lower normalized mean-squared error than the n-step prediction agent. Finally, we show the importance of learning in deployment, by comparing a TD agent trained purely offline with no online updating to a TD agent that learns online. This final result is one of the first to motivate the importance of adapting predictions in real-time, for non-stationary high-volume systems in the real world.

show abstract

“…torque on its joints, to video compression (Mandhane et al, 2022), to maximizing the efficiency of commercial cooling systems under stability constraints (Luo et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs

Moskovitz¹,

O’Donoghue²,

Veeriah³

et al. 2023

Preprint

View full text Add to dashboard Cite

In recent years, Reinforcement Learning (RL) has been applied to real-world problems with increasing success. Such applications often require to put constraints on the agent's behavior. Existing algorithms for constrained RL (CRL) rely on gradient descent-ascent, but this approach comes with a caveat. While these algorithms are guaranteed to converge on average, they do not guarantee last-iterate convergence, i.e., the current policy of the agent may never converge to the optimal solution. In practice, it is often observed that the policy alternates between satisfying the constraints and maximizing the reward, rarely accomplishing both objectives simultaneously. Here, we address this problem by introducing Reinforcement Learning with Optimistic Ascent-Descent (ReLOAD), a principled CRL method with guaranteed lastiterate convergence. We demonstrate its empirical effectiveness on a wide variety of CRL problems including discrete MDPs and continuous control. In the process we establish a benchmark of challenging CRL problems.

show abstract

Controlling Commercial Cooling Systems Using Reinforcement Learning

Cited by 3 publications

References 0 publications

Una introducciön amable pero rigurosa al aprendizaje por refuerzo

Una introducciön amable pero rigurosa al aprendizaje por refuerzo

GVFs in the real world: making predictions online for water treatment

ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs

Contact Info

Product

Resources

About