Double Q-learning for Demand Response of an Electric Water Heater

Peirelinck, Thijs; Hermans, C.A.M.; Spiessens, Fred

doi:10.1109/isgteurope.2019.8905776

“…The bars depict the 95 % confidence bound (over 20 simulation runs). As known from previous research [1,16,17], RL (without pre-training) manages to reduce cost compared to a hysteresis controller. Here, cost has been reduced by 23.4 %.…”

Section: A Sinusoidal Price Profilementioning

confidence: 95%

“…The stratified buffer model we use has been presented and validated in the lab by Vanthournout et al [14]. This model has been used in DR research several times [1,16,17], and for more details we refer to Vanthournout et al [14].…”

Section: B Two Mass Buffer Modelmentioning

confidence: 99%

“…Demand response of an EWH gives rise to a Sequential Decision Problem (SDP). In earlier work [1,3,16,17] the learning agent's SDP has already been formulated as a Partially Observable Markov Decision Process (POMDP). This section reviews the POMDP and introduces the algorithms used in our work.…”

Section: Problem Formulation and Algorithmsmentioning

confidence: 99%

“…3) Cost-function: We consider an energy arbitrage DR setting, with Time-of-Use (ToU) pricing. The cost-function defines the MDP's reward as in (17), given energy price λ t and energy consumption E t = H(u t )P r ∆t.…”

Section: A Markov Decision Processmentioning

confidence: 99%

See 2 more Smart Citations

Domain Randomization for Demand Response of an Electric Water Heater

Peirelinck

¹

,

Hermans

²

,

Spiessens

³

2021

IEEE Trans. Smart Grid

Self Cite

View full text Add to dashboard Cite

Thermostatically Controlled Loads (TCLs) provide a source of demand flexibility, and are often considered a good source for Demand Response (DR) applications. Due to their heterogeneity, and as such a lack of dynamics models, Reinforcement Learning (RL) is often used to exploit this flexibility. Unfortunately, RL requires exploratory interaction with the TCL, resulting in a period of potential discomfort for the users. We present an approach to reduce this exploratory time by pretraining the RL-agent. Domain randomization is used to facilitate knowledge transfer. We evaluate the pre-training potential in a DR energy arbitrage scenario with an Electric Water Heater (EWH). Our experiments show that a priori knowledge about EWH dynamics can be used to initialize and improve the control policy. In our experiments, pre-training attributes to 8.8 % additional cost savings, compared to starting from scratch.

show abstract

“…The stratified buffer model we use has been presented and validated in the lab by Vanthournout et al [14]. This model has been used in DR research several times [1,16,17], and for more details we refer to Vanthournout et al [14].…”

Section: Stratified Buffer Modelmentioning

confidence: 99%

Domain Randomization for Demand Response of an Electric Water Heater

Peirelinck

¹

,

Hermans²,

Spiessens³

2021

2021 IEEE Power &Amp; Energy Society General Meeting (PESGM)

Self Cite

View full text Add to dashboard Cite

Thermostatically Controlled Loads (TCLs) provide a source of demand flexibility, and are often considered a good source for Demand Response (DR) applications. Due to their heterogeneity, and as such a lack of dynamics models, Reinforcement Learning (RL) is often used to exploit this flexibility. Unfortunately, RL requires exploratory interaction with the TCL, resulting in a period of potential discomfort for the users. We present an approach to reduce this exploratory time by pretraining the RL-agent. Domain randomization is used to facilitate knowledge transfer. We evaluate the pre-training potential in a DR energy arbitrage scenario with an Electric Water Heater (EWH). Our experiments show that a priori knowledge about EWH dynamics can be used to initialize and improve the control policy. In our experiments, pre-training attributes to 8.8 % additional cost savings, compared to starting from scratch.

show abstract