A Knowledge-based reinforcement learning control approach using deep Q network for cooling tower in HVAC systems

Yu, Zujiang; Xu, Yang; Gao, Feng; Huang, Jian; Tu, Rang; Cui, Jiarui

doi:10.1109/cac51589.2020.9327385

Cited by 4 publications

(1 citation statement)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An article was excluded if it did not provide sufficient details to determine whether the action space was binary, discrete or continuous [31][32][33][34].…”

mentioning

confidence: 99%

A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems

2022

View full text Add to dashboard Cite

Reinforcement learning has emerged as a potentially disruptive technology for control and optimization of HVAC systems. A reinforcement learning agent takes actions, which can be direct HVAC actuator commands or setpoints for control loops in building automation systems. The actions are taken to optimize one or more targets, such as indoor air quality, energy consumption and energy cost. The agent receives feedback from the HVAC systems to quantify how well these targets have been achieved. The feedback is captured by a reward function designed by the developer of the reinforcement learning agent. A few reviews have focused on the reward aspect of reinforcement learning applications for HVAC. However, there is a lack of reviews that assess how the actions of the reinforcement learning agent have been formulated, and how this impacts the possibilities to achieve various optimization targets in single zone or multi-zone buildings. The aim of this review is to identify the action formulations in the literature and to assess how the choice of formulation impacts the level of abstraction at which the HVAC systems are considered. Our methodology involves a search string in the Web of Science database and a list of selection criteria applied to each article in the search results. For each selected article, a three-tier categorization of the selected articles has been performed. Firstly, the applicability of the approach to buildings with one or more zones is considered. Secondly, the articles are categorized by the type of action taken by the agent, such as a binary, discrete or continuous action. Thirdly, the articles are categorized by the aspects of the indoor environment being controlled, namely temperature, humidity or air quality. The main result of the review is this three-tier categorization that reveals the community’s emphasis on specific HVAC applications, as well as the readiness to interface the reinforcement learning solutions to HVAC systems. The article concludes with a discussion of trends in the field as well as challenges that require further research.

show abstract

“…An article was excluded if it did not provide sufficient details to determine whether the action space was binary, discrete or continuous [31][32][33][34].…”

mentioning

confidence: 99%

A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems

2022

View full text Add to dashboard Cite

show abstract

A Survey on the Control Techniques used In Process Control

2023

2023 International Conference on Innovations in Engineering and Technology (ICIET)

View full text Add to dashboard Cite

Deep Forest-Based DQN for Cooling Water System Energy Saving Control in HVAC

Han

Chen

et al. 2022

Buildings

View full text Add to dashboard Cite

Currently, reinforcement learning (RL) has shown great potential in energy saving in HVAC systems. However, in most cases, RL takes a relatively long period to explore the environment before obtaining an excellent control policy, which may lead to an increase in cost. To reduce the unnecessary waste caused by RL methods in exploration, we extended the deep forest-based deep Q-network (DF-DQN) from the prediction problem to the control problem, optimizing the running frequency of the cooling water pump and cooling tower in the cooling water system. In DF-DQN, it uses the historical data or expert experience as a priori knowledge to train a deep forest (DF) classifier, and then combines the output of DQN to attain the control frequency, where DF can map the original action space of DQN to a smaller one, so DF-DQN converges faster and has a better energy-saving effect than DQN in the early stage. In order to verify the performance of DF-DQN, we constructed a cooling water system model based on historical data. The experimental results show that DF-DQN can realize energy savings from the first year, while DQN realized savings from the third year. DF-DQN’s energy-saving effect is much better than DQN in the early stage, and it also has a good performance in the latter stage. In 20 years, DF-DQN can improve the energy-saving effect by 11.035% on average every year, DQN can improve by 7.972%, and the model-based control method can improve by 13.755%. Compared with traditional RL methods, DF-DQN can avoid unnecessary waste caused by exploration in the early stage and has a good performance in general, which indicates that DF-DQN is more suitable for engineering practice.

show abstract

A Knowledge-based reinforcement learning control approach using deep Q network for cooling tower in HVAC systems

Cited by 4 publications

References 20 publications

A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems

A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems

A Survey on the Control Techniques used In Process Control

Deep Forest-Based DQN for Cooling Water System Energy Saving Control in HVAC

Contact Info

Product

Resources

About