Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking

Pardo, Fabio

doi:10.48550/arxiv.2011.07537

Cited by 7 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Figure 4 presents the training curves of the each of the reduced rule sets, and Figure 5 shows the training curves of the rest of the models. For comparisons with different reinforcement learning algorithms in the standard AntBullet environment see Pardo [38] for baselines where, e.g., Proximal Policy Optimization (PPO) achieves a score of around 3100, Deep Deterministic Policy Gradient (DDPG) scores around 2500, and Advantage Actor-Critic (A2C) scores around 1800.…”

Section: Resultsmentioning

confidence: 99%

Evolving and merging hebbian learning rules

Winther

Risi

2021

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

Generalization to out-of-distribution (OOD) circumstances after training remains a challenge for artificial agents. To improve the robustness displayed by plastic Hebbian neural networks, we evolve a set of Hebbian learning rules, where multiple connections are assigned to a single rule. Inspired by the biological phenomenon of the genomic bottleneck, we show that by allowing multiple connections in the network to share the same local learning rule, it is possible to drastically reduce the number of trainable parameters, while obtaining a more robust agent. During evolution, by iteratively using simple K-Means clustering to combine rules, our Evolve & Merge approach is able to reduce the number of trainable parameters from 61,440 to 1,920, while at the same time improving robustness, all without increasing the number of generations used. While optimization of the agents is done on a standard quadruped robot morphology, we evaluate the agents' performances on slight morphology modifications in a total of 30 unseen morphologies. Our results add to the discussion on generalization, overfitting and OOD adaptation. To create agents that can adapt to a wider array of unexpected situations, Hebbian learning combined with a regularising "genomic bottleneck" could be a promising research direction. CCS CONCEPTS• Computing methodologies → Bio-inspired approaches.

show abstract

Section: Resultsmentioning

confidence: 99%

Evolving and merging hebbian learning rules

Winther

Risi

2021

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

show abstract

“…Libraries Neural networks were implemented in PyTorch (Paszke et al, 2019). The RL algorithms were implemented using Tonic (Pardo, 2021). The ES algorithm was implemented using ES Torch (Karakasli, 2020).…”

Section: A Experimental Detailsmentioning

confidence: 99%

Neural Circuit Architectural Priors for Embodied Control

Bhattasali¹,

Zador²,

Engel³

2022

Preprint

View full text Add to dashboard Cite

Artificial neural networks for simulated motor control and robotics often adopt generic architectures like fully connected MLPs. While general, these tabula rasa architectures rely on large amounts of experience to learn, are not easily transferable to new bodies, and have internal dynamics that are difficult to interpret. In nature, animals are born with highly structured connectivity in their nervous systems shaped by evolution; this innate circuitry acts synergistically with learning mechanisms to provide inductive biases that enable most animals to function well soon after birth and improve abilities efficiently. Convolutional networks inspired by visual circuitry have encoded useful biases for vision. However, it is unknown the extent to which ANN architectures inspired by neural circuitry can yield useful biases for other domains. In this work, we ask what advantages biologically inspired network architecture can provide in the context of motor control. Specifically, we translate C. elegans circuits for locomotion into an ANN model controlling a simulated Swimmer agent. On a locomotion task, our architecture achieves good initial performance and asymptotic performance comparable with MLPs, while dramatically improving data efficiency and requiring orders of magnitude fewer parameters. Our architecture is more interpretable and transfers to new body designs. An ablation analysis shows that principled excitation/inhibition is crucial for learning, while weight initialization contributes to good initial performance. Our work demonstrates several advantages of ANN architectures inspired by systems neuroscience and suggests a path towards modeling more complex behavior.

show abstract

“…Dopamine [4] is focusing on DQN variants [28,6,19] to make them available for researchers as baseline implementations. Tonic [31] aims to provide many of continuous control algorithms with large-scale benchmark results. There are two design differences between d3rlpy and the existing libraries.…”

Section: Related Workmentioning

confidence: 99%

“…Moreover, the non-standardized implementations makes it difficult for researchers to track the exact implementation difference between algorithms. On the other hand, there are already many libraries that provide a collection of deep RL algorithms [15,4,31]. However, they are designed for online RL paradigms and not providing complete supports for offline RL in terms of algorithms and interfaces.…”

Section: Introductionmentioning

confidence: 99%

d3rlpy: An Offline Deep Reinforcement Learning Library

Seno

Imai

2021

Preprint

View full text Add to dashboard Cite

In this paper, we introduce d3rlpy, an open-sourced offline deep reinforcement learning (RL) library for Python. d3rlpy supports a number of offline deep RL algorithms as well as online algorithms via a user-friendly API. To assist deep RL research and development projects, d3rlpy provides practical and unique features such as data collection, exporting policies for deployment, preprocessing and postprocessing, distributional Q-functions, multi-step learning and a convenient command-line interface. Furthermore, d3rlpy additionally provides a novel graphical interface that enables users to train offline RL algorithms without coding programs. Lastly, the implemented algorithms are benchmarked with D4RL datasets to ensure the implementation quality. The d3rlpy source code can be found on GitHub: https://github.com/takuseno/d3rlpy.

show abstract

Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking

Cited by 7 publications

References 0 publications

Evolving and merging hebbian learning rules

Evolving and merging hebbian learning rules

Neural Circuit Architectural Priors for Embodied Control

d3rlpy: An Offline Deep Reinforcement Learning Library

Contact Info

Product

Resources

About