Reinforcement Learning of Heuristic EV Fleet Charging in a Day-Ahead Electricity Market

Abstract

Abstract-This paper addresses the problem of defining a day-ahead consumption plan for charging a fleet of electric vehicles (EVs), and following this plan during operation. A challenge herein is the beforehand unknown charging flexibility of EVs, which depends on numerous details about each EV (e.g., plug-in times, power limitations, battery size, power curve, etc.). To cope with this challenge, EV charging is controlled during opertion by a heuristic scheme, and the resulting charging behavior of the EV fleet is learned by using batch mode reinforcement learning. Based on this learned behavior, a cost-effective day-ahead consumption plan can be defined. In simulation experiments, our approach is benchmarked against a multistage stochastic programming solution, which uses an exact model of each EVs charging flexibility. Results show that our approach is able to find a day-ahead consumption plan with comparable quality to the benchmark solution, without requiring an exact day-ahead model of each EVs charging flexibility.Index Terms-Demand-side management, electric vehicles (EVs), reinforcement learning (RL), stochastic programming (SP). NOMENCLATUREThe symbols and notations used throughout this paper are summarized below. Manuscript received October 15, 2013; revised March 20, 2014 and August 28, 2014; accepted October 20, 2014 Offset from day-ahead prices, to define imbalance prices. Functions T(h)Mapping from market period h to the set of control periods in market period h. f heur Heuristic function to dispatch power to EVs. Real variables E da hEnergy bought in the day-ahead market for market period h. Energy charged by the EV fleet in control period t. i P n t Charging power of EV i during control period t in senario n.

Definition and Evaluation of Model-Free Coordination of Electrical Vehicle Charging With Reinforcement Learning

“…Note that our work is different from [8] and [10] in two aspects: (i) unlike [8] and [10], our proposed approach does not take the control decisions in separate steps (i.e., taking aggregate energy consumption in one step and coordinating individual EV charging in a second step to meet the already decided energy consumption) and instead it takes decisions directly and jointly for all individual EVs using an efficient representation of an aggregate state of a group of EVs, hence (ii) our approach does not need a heuristic algorithm, but instead learns the aggregate load while finding an optimum policy to flatten the load curve. We now describe our MDP model, and subsequently the batch reinforcement learning approach to train it.…”

Multi-agent modeling and analysis of EV users’ travel willingness based on an integrated causal/statistical/behavioral model

“…Multi-agents are usually generated from optimization models by using expert knowledge and intelligent learning methods [23]. If the multi-agents are only demanded to imitate the behaviors of participants, they can be directly generated from one or more types of sources in the behavioral/statistical/causal data.…”

Reinforcement Learning for Electric Power System Decision and Control: Past Considerations and Perspectives

“…Power system components considered include: dynamic brake Ernst et al (2004); Glavic (2005), thyristor controlled series capacitor Ernst et al (2004Ernst et al ( , 2009, quadrature booster Li and Wu (1999), synchronous generators (all AGC related references), individual or aggregated loads Vandael et al (2015); Ruelens et al (2016), etc. If used as a multi-agent system, then additional state variables must be introduced to ensure convergence of these essentially distributed computation schemes, and an adapted variant of standard RL methods is often used (for example correlated equilibrium Q(λ) Yu et al (2012a)).…”