Local Energy Trading Behavior Modeling With Deep Reinforcement Learning

Chen, Tao; Su, Wencong

doi:10.1109/access.2018.2876652

Cited by 92 publications

(51 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As an extension of Q-learning on multi-dimensional continuous state space, authors in [25], [26] proposed the deep Q network (DQN) method which employs a DNN to approximate the action-value function, and has performed at the level of expert humans in playing Atari 2600 games. Inspired by this pioneering work, several recent papers have employed the DQN method to various smart grid applications such as voltage control [27], residential load control [28], building energy management systems [29], electric vehicles [30] and energy storage scheduling [31] and energy trading for prosumers [32]. However, although previous work has demonstrated high quality performance of the DQN method in problems with continuous state spaces, its performance in problems with continuous action spaces is less satisfactory because the employed DNN is trained to produce discrete action-value estimates rather than continuous actions [33], which significantly hinders its effectiveness in addressing the examined market modeling problem, since market players' actions are continuous and multi-dimensional.…”

Section: A Background and Motivationmentioning

confidence: 99%

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

Qiu

Sun

et al. 2020

IEEE Trans. Smart Grid

181

View full text Add to dashboard Cite

Dawei Qiu (S'18) is currently pursuing the Ph.D. degree at Imperial College London, London, U.K. His current research interests include game-theoretic and agent-based modeling in wholesale as well as retail electricity markets. Mingyang Sun (M'16) received the Ph.D. degree from Imperial College London, London, U.K., in 2017. He is currently a Research Associate in this institution. His current research interests include big data analytics and artificial intelligence in energy systems. Dimitrios Papadaskalopoulos (M'13) is a Research Fellow at Imperial College London, London, U.K. His current research focuses on the development and application of distributed and market-based approaches for the coordination of operation and planning decisions in power systems, employing optimization and game theoretic principles. Goran Strbac (M'95) is a Professor of Electrical Energy Systems at Imperial College London, London, U.K. His research interests include electricity system operation, investment and pricing, and integration of renewable generation and distributed energy resources.

show abstract

Section: A Background and Motivationmentioning

confidence: 99%

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

Qiu

Sun

et al. 2020

IEEE Trans. Smart Grid

181

View full text Add to dashboard Cite

show abstract

“…To cope with the emergency control in the power system, a novel adaptive strategy and an opensource platform were designed for the grid control [21]. Reference [25] proposed a deep Q-learning based algorithm for local energy trading to optimize the decision-making processes of prosumers. In [22], a novel two-timescale voltage regulation scheme was introduced for smart grids by coupling deep Q-learning with physics-based optimization.…”

Section: B Drl-based Power System Managementmentioning

confidence: 99%

“…Unlike traditional reinforcement learning, the DRL algorithms use powerful deep neuron networks to approximate their value function (such as Q-table), enabling automatic high-dimensional feature extraction and end-toend learning. Recently, the advantages of DRL were recognized by the community and some attempts were made to leverage DRL in various applications for electrical grid, including operational control [21]- [24], electricity market [25], [26], demand response [27] and energy management [28]. Although these applications presented advantageous results in their respective fields, several challenges were encountered.…”

mentioning

confidence: 99%

“…Although these applications presented advantageous results in their respective fields, several challenges were encountered. Some early works [21], [22], [25] adopted deep Qnetwork as the agent's policy, which could only deal with control problems with discrete action spaces. These applications were not practical for physical regulation tasks in power systems, requiring continuous action spaces.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Probability-Based Energy Reinforced Management of Electric Vehicle Aggregation in the Electrical Grid Frequency Regulation

et al. 2020

View full text Add to dashboard Cite

The model uncertainties and the heterogeneous energy states burden the effective aggregation of electric vehicles (EVs), especially coupling with the real-time frequency dynamic of the electrical grid. Integrating the advantages of deep learning and reinforcement learning, deep reinforcement learning shows its potential to relieve this challenge, where an intelligent agent fully considers the individual state of charge (SOC) difference of EV and the grid state to optimize the aggregation performance. However, existing policies of deep reinforcement learning usually provide deterministic and certain actions, and it is difficult to deal with the increasing uncertainties and randomness in modern electrical systems. In this paper, a probability-based management strategy is proposed with continuous action space based on the deep reinforcement learning, which provides fine-grained energy management and addresses the time-varying dynamics from EVs and electrical grid simultaneously. Moreover, an optimization based on the proximal policy is further introduced to clip the policy upgradation speed to enhance the training stability. The effectiveness of proposed energy management structure and policy optimization strategy are verified on various scenarios and uncertainties, which demonstrates advantageous performance in the SOC management and frequency maintenance. Besides the performance merits, the training procedure is also presented revealing the evolution reason for the proposed approach. INDEX TERMS State of charge (SOC), deep reinforcement learning, hybrid framework, electric vehicle aggregator, multiple-input and multiple-output. Nomenclature

show abstract

“…In [8], Chen et al proposed an innovative method based on the concept of 'energy trading'. Through an ad-hoc mathematical model of energy trading strategies of a prosumer in the proposed holistic market model, the prosumer's decision-making process will be analyzed as a Markov decision process so that the local market participation will be solved by deep reinforcement learning technology with experience replay mechanism.…”

Section: Deep Learning (Lstm) and Rl Based Trading Systems: Literaturmentioning

confidence: 99%

Deep LSTM with Reinforcement Learning Layer for Financial Trend Prediction in FX High Frequency Trading Systems

Rundo

2019

Applied Sciences

View full text Add to dashboard Cite

High-frequency trading is a method of intervention on the financial markets that uses sophisticated software tools, and sometimes also hardware, with which to implement high-frequency negotiations, guided by mathematical algorithms, that act on markets for shares, options, bonds, derivative instruments, commodities, and so on. HFT strategies have reached considerable volumes of commercial traffic, so much so that it is estimated that they are responsible for most of the transaction traffic of some stock exchanges, with percentages that, in some cases, exceed 70% of the total. One of the main issues of the HFT systems is the prediction of the medium-short term trend. For this reason, many algorithms have been proposed in literature. The author proposes in this work the use of an algorithm based both on supervised Deep Learning and on a Reinforcement Learning algorithm for forecasting the short-term trend in the currency FOREX (FOReign EXchange) market to maximize the return on investment in an HFT algorithm. With an average accuracy of about 85%, the proposed algorithm is able to predict the medium-short term trend of a currency cross based on the historical trend of this and by means of correlation data with other currency crosses using techniques known in the financial field with the term arbitrage. The final part of the proposed pipeline includes a grid trading engine which, based on the aforementioned trend predictions, will perform high frequency operations in order to maximize profit and minimize drawdown. The trading system has been validated over several financial years and on the EUR/USD cross confirming the high performance in terms of Return of Investment (98.23%) in addition to a reduced drawdown (15.97 %) which confirms its financial sustainability.

show abstract

Local Energy Trading Behavior Modeling With Deep Reinforcement Learning

Cited by 92 publications

References 18 publications

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

Deep Reinforcement Learning for Strategic Bidding in Electricity Markets

Probability-Based Energy Reinforced Management of Electric Vehicle Aggregation in the Electrical Grid Frequency Regulation

Deep LSTM with Reinforcement Learning Layer for Financial Trend Prediction in FX High Frequency Trading Systems

Contact Info

Product

Resources

About