Distributional reinforcement learning with unconstrained monotonic neural networks

Théate, Thibaut; Wehenkel, Antoine; Bolland, Adrien; Louppe, Gilles; Ernst, Damien

doi:10.1016/j.neucom.2023.02.049

Cited by 3 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, it ensures the accurate estimation of the risk, as defined in Section 3.2.2. This observation is in line with the findings of the research paper [27], introducing the UMDQN algorithm, and suggests that the solution introduced to achieve risk-sensitivity does not significantly alter the properties of the original distributional RL algorithm. Secondly, as previously explained, Figure 6 highlights the relevance of each function introduced (Q π , R π and U π ) for making and motivating a decision.…”

Section: Probability Distribution Visualisationsupporting

confidence: 88%

“…The distributional RL algorithm selected to assess the soundness of the methodology introduced to learn risk-sensitive decision-making policies is the Unconstrained Monotonic Deep Q-Network with Cramer (UMDQN-C) [27]. Basically, this particular distributional RL algorithm models the CDF of the random return in a continuous way by taking advantage of the Cramer distance to derive the TD-error.…”

Section: Risk-sensitive Distributional Rl Algorithm Analysedmentioning

confidence: 99%

“…This second feature clearly motivated the selection of this specific distributional RL algorithm to conduct the following experiments, since accurate random return probability distributions are required to properly estimate the risk. The reader can refer to the original research paper [27] for more information about the UMDQN-C distributional RL algorithm.…”

Section: Risk-sensitive Distributional Rl Algorithm Analysedmentioning

confidence: 99%

“…Algorithm A1 presents the novel Risk-Sensitive Unconstrained Monotonic Deep Q-Network with Cramer algorithm (RS-UMDQN-C). Basically, this new RL algorithm is the result of the application of the methodology described in Algorithm 1 to the UMDQN-C algorithm thoroughly introduced in [27].…”

Section: Conflicts Of Interestmentioning

confidence: 99%

See 3 more Smart Citations

Risk-Sensitive Policy with Distributional Reinforcement Learning

Théate

Ernst

2023

Algorithms

Self Cite

View full text Add to dashboard Cite

Classical reinforcement learning (RL) techniques are generally concerned with the design of decision-making policies driven by the maximisation of the expected outcome. Nevertheless, this approach does not take into consideration the potential risk associated with the actions taken, which may be critical in certain applications. To address that issue, the present research work introduces a novel methodology based on distributional RL to derive sequential decision-making policies that are sensitive to the risk, the latter being modelled by the tail of the return probability distribution. The core idea is to replace the Q function generally standing at the core of learning schemes in RL by another function, taking into account both the expected return and the risk. Named the risk-based utility function U, it can be extracted from the random return distribution Z naturally learnt by any distributional RL algorithm. This enables the spanning of the complete potential trade-off between risk minimisation and expected return maximisation, in contrast to fully risk-averse methodologies. Fundamentally, this research yields a truly practical and accessible solution for learning risk-sensitive policies with minimal modification to the distributional RL algorithm, with an emphasis on the interpretability of the resulting decision-making process.

show abstract

Section: Probability Distribution Visualisationsupporting

confidence: 88%

Section: Risk-sensitive Distributional Rl Algorithm Analysedmentioning

confidence: 99%

Section: Risk-sensitive Distributional Rl Algorithm Analysedmentioning

confidence: 99%

Section: Conflicts Of Interestmentioning

confidence: 99%

See 2 more Smart Citations