Memory-based Deep Reinforcement Learning for POMDPs

Meng, Lingheng; Gorbet, Rob; Kulić, Dana

doi:10.1109/iros51168.2021.9636140

Cited by 47 publications

(52 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conducted experiments on the Pendulum control problem, where the pendulum is learned to swing up and stay upright during every episode. Unlike the original fully-observable version, each dimension of every state is converted to zero with probability p miss = 0.1 when the agent receives observation (Meng, Gorbet, and Kulic 2021). This random missing setting induces partial observability and makes a simple control problem challenging.…”

Section: Pendulum -Random Missing Versionmentioning

confidence: 99%

“…The information from the past should be extracted and exploited during the learning phase to compensate for the information loss due to partial observability. Partially observable situations are prevalent in real-world problems such as control tasks when observations are noisy, some part of the underlying state information is deleted, or long-term information needs to be estimated (Han, Doya, and Tani 2020b;Meng, Gorbet, and Kulic 2021).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning

Park

Choi

Sung

2022

AAAI

View full text Add to dashboard Cite

This paper proposes a new sequential model learning architecture to solve partially observable Markov decision problems. Rather than compressing sequential information at every timestep as in conventional recurrent neural network-based methods, the proposed architecture generates a latent variable in each data block with a length of multiple timesteps and passes the most relevant information to the next block for policy optimization. The proposed blockwise sequential model is implemented based on self-attention, making the model capable of detailed sequential learning in partial observable settings. The proposed model builds an additional learning network to efficiently implement gradient estimation by using self-normalized importance sampling, which does not require the complex blockwise input data reconstruction in the model learning. Numerical results show that the proposed method significantly outperforms previous methods in various partially observable environments.

show abstract

Section: Pendulum -Random Missing Versionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning

Park

Choi

Sung

2022

AAAI

View full text Add to dashboard Cite

show abstract

“…Hausknecht & Stone (2015) extended the Deep Q-Network (DQN) algorithm and examined the way how episodes and their hidden states are sampled from the experience replay buffer. More literature is available on adding recurrence to variants of Deep Deterministic Policy Gradient (DDPG) algorithms (Heess et al, 2015;Meng et al, 2021). Especially Ni et al (2021) proposed a recurrent Twin-Delayed DDPG implementation and shared how their replay buffer is made more memory-efficient.…”

Section: Applications Of Recurrent Layers In Drlmentioning

confidence: 99%

Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization

Pleines¹,

Pallasch²,

Zimmer³

et al. 2022

Preprint

View full text Add to dashboard Cite

At first sight it may seem straightforward to use recurrent layers in Deep Reinforcement Learning algorithms to enable agents to make use of memory in the setting of partially observable environments. Starting from widely used Proximal Policy Optimization (PPO), we highlight vital details that one must get right when adding recurrence to achieve a correct and efficient implementation, namely: properly shaping the neural net's forward pass, arranging the training data, correspondingly selecting hidden states for sequence beginnings and masking paddings for loss computation. We further explore the limitations of recurrent PPO by benchmarking the contributed novel environments Mortar Mayhem and Searing Spotlights that challenge the agent's memory beyond solely capacity and distraction tasks. Remarkably, we can demonstrate a transition to strong generalization in Mortar Mayhem when scaling the number of training seeds, while the agent does not succeed on Searing Spotlights, which seems to be a tough challenge for memory-based agents.

show abstract

“…We conducted experiments on the Pendulum control problem (Brockman et al 2016), where the pendulum is learned to swing up and stay upright during every episode. Unlike the original fully-observable version, each dimension of every state is converted to zero with probability p miss = 0.1 when the agent receives observation (Meng, Gorbet, and Kulic 2021). This random missing setting induces partial observability and makes a simple control problem challenging.…”

Section: Pendulum -Random Missing Versionmentioning

confidence: 99%

“…We focused on the performance comparison within a fixed sample size rather than the average runtime or estimated energy cost. We used the code of Mountain Hike environment from (Igl et al 2018), and we modified the POMDP wrapper code provided by (Meng, Gorbet, and Kulic 2021) for the considered Pendulum environment. The episode length is 200 timesteps in the considered Pendulum, and the maximum episode length is 128 timesteps in the sequential target-reaching task.…”

Section: Details In Implementationmentioning

confidence: 99%