Risk-sensitive reinforcement learning

Vadori, Nelson; Ganesh, Sumitra; Reddy, P. Kiran Kumar; Veloso, Manuela

doi:10.1145/3383455.3422519

Cited by 9 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Risk-sensitive RL: In the field of risk-sensitive RL, traditional research has focused on optimizing RL agents for specific risk measures (Howard and Matheson 1972;Sato, Kimura, and Kobayashi 2001;Mihatsch and Neuneier 2002;Tamar, Glassner, and Mannor 2015;Chow et al 2017;Dabney et al 2018;Vadori et al 2020). Early efforts include optimizing an instance of WV@R like worst-case (Mihatsch and Neuneier 2002) or CV@R (Tamar, Glassner, and Mannor 2015;Chow et al 2017;Dabney et al 2018).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures

Yoo,

Park,

Woo

2024

AAAI

View full text Add to dashboard Cite

In application domains requiring mission-critical decision making, such as finance and robotics, the optimal policy derived by reinforcement learning (RL) often hinges on a preference for risk management. Yet, the dynamic nature of risk measures poses considerable challenges to achieving generalization and adaptation of risk-sensitive policies in the context of RL. In this paper, we propose a risk-conditioned RL model that enables rapid policy adaptation to varying risk measures via a unified risk representation, the Weighted Value-at-Risk (WV@R). To sample risk measures that avoid undue optimism, we construct a risk proposal network employing a conditional adversarial auto-encoder and a normalizing flow. This network establishes coherent representations for risk measures, preserving the continuity in terms of the Wasserstein distance on the risk measures. The normalizing flow is used to support non-crossing quantile regression that obtains valid samples for risk measures, and it is also applied to the agent’s critic to ascertain the preservation of monotonicity in quantile estimations. Through experiments with locomotion, finance, and self-driving scenarios, we show that our model is capable of adapting to a range of risk measures, achieving comparable performance to the baseline models individually trained for each measure. Our model often outperforms the baselines, especially in the cases when exploration is required during training but risk-aversion is favored during evaluation.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Using the normalizing flow, we also addressed the crossing quantile problem. Our framework focuses on WV@R, while not encompassing all risk measures, such as CMV (Vadori et al 2020) or variance, but can cover a wide range of risk measures applicable to real-world problems.…”

Section: Conclusion and Limitationmentioning

confidence: 99%

Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures

Yoo,

Park,

Woo

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…It implements DRL (Deep Reinforcement Learning) algorithms to simulate a wide array of markets as well as trading constraints to decide where to trade, at what price and what quantity [91]. DRL addresses the dynamic decision-making problems by offering portfolio scalability and market model independence [92][93][94][95][96]. This gives it a competitive edge over human traders [97][98].…”

Section: Trading and Financementioning

confidence: 99%

Reinforcement Learning Review: Past Acts, Present Facts and Future Prospects

Kommey,

Isaac,

Tamakloe

et al. 2024

ITJRD

View full text Add to dashboard Cite

Reinforcement Learning (RL) is fast gaining traction as a major branch of machine learning, its applications have expanded well beyond its typical usage in games. Several subfields of reinforcement learning like deep reinforcement learning and multi-agent reinforcement learning are also expanding rapidly. This paper provides an extensive review on the field from the point of view of Machine Learning (ML). It begins by providing a historical perspective on the field then proceeds to lay a theoretical background on the field. It further discusses core reinforcement learning problems and approaches taken by different subfields before discussing the state of the art in the field. An inexhaustive list of applications of reinforcement learning is provided and their practicability and scalability assessed. The paper concludes by highlighting some open areas or issues in the field

show abstract

“…Vadori et al. (2020) developed a martingale approach to learn policies that are sensitive to the uncertainty of the rewards and are meaningful under some market scenarios. Another line of work focuses on constrained RL problems with different risk criteria (Achiam et al., 2017; Chow et al., 2017, 2015; Ding et al., 2021; Tamar et al., 2015; Zheng & Ratliff, 2020).…”

Section: Further Developments For Mathematical Finance and Reinforcem...mentioning

confidence: 99%

Recent advances in reinforcement learning in finance

Hambly

Yang

2023

Mathematical Finance

View full text Add to dashboard Cite

The rapid changes in the finance industry due to the increasing amount of data have revolutionized the techniques on data processing and data analysis and brought new theoretical and computational challenges. In contrast to classical stochastic control theory and other analytical approaches for solving financial decision‐making problems that heavily reply on model assumptions, new developments from reinforcement learning (RL) are able to make full use of the large amount of financial data with fewer model assumptions and to improve decisions in complex financial environments. This survey paper aims to review the recent developments and use of RL approaches in finance. We give an introduction to Markov decision processes, which is the setting for many of the commonly used RL approaches. Various algorithms are then introduced with a focus on value‐ and policy‐based methods that do not require any model assumptions. Connections are made with neural networks to extend the framework to encompass deep RL algorithms. We then discuss in detail the application of these RL algorithms in a variety of decision‐making problems in finance, including optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo‐advising. Our survey concludes by pointing out a few possible future directions for research.

show abstract

Risk-sensitive reinforcement learning

Cited by 9 publications

References 11 publications

Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures

Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures

Reinforcement Learning Review: Past Acts, Present Facts and Future Prospects

Recent advances in reinforcement learning in finance

Contact Info

Product

Resources

About