AI Safety Gridworlds

Leike, Jan; Martic, Miljan; Krakovna, Victoria; Ortega, Pedro A.; Everitt, Tom; Lefrancq, Andrew; Orseau, Laurent; Legg, Shane

doi:10.48550/arxiv.1711.09883

Cited by 65 publications

(107 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is particularly concerning given that Silver et al are highly influential researchers and employed at DeepMind, one of the organisations best equipped to expand the frontiers of AGI. While Silver et al "hope that other researchers will join us on our quest", we instead hope that the creation of AGI based on reward maximisation is tempered by other researchers with an understanding of the issues of AI safety [45,47] and an appreciation of the benefits of multi-objective agents [1,2].…”

Section: Discussionmentioning

confidence: 99%

Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)

Vamplew¹,

Smith²,

Källström³

et al. 2021

Preprint

View full text Add to dashboard Cite

The recent paper "Reward is Enough" by Silver, Singh, Precup and Sutton posits that the concept of reward maximisation is sufficient to underpin all intelligence, both natural and artificial. We contest the underlying assumption of Silver et al. that such reward can be scalar-valued. In this paper we explain why scalar rewards are insufficient to account for some aspects of both biological and computational intelligence, and argue in favour of explicitly multi-objective models of reward maximisation. Furthermore, we contend that even if scalar reward functions can trigger intelligent behaviour in specific cases, it is still undesirable to use this approach for the development of artificial general intelligence due to unacceptable risks of unsafe or unethical behaviour.

show abstract

Section: Discussionmentioning

confidence: 99%

Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)

Vamplew¹,

Smith²,

Källström³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Leike et al [134] introduced a suite of reinforcement learning environments to measure the agent's compliance with the intended safe behavior. Their work categorizes AI safety problems into two areas of specification and robustness, which cover various problems, including avoiding side effects and safe exploration, and robustness to self-modification and distributional shift.…”

Section: Ai Safetymentioning

confidence: 99%

Taxonomy of Machine Learning Safety: A Survey and Primer

Mohseni¹,

Wang²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

The open-world deployment of Machine Learning (ML) algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities such as interpretability, verifiability, and performance limitations. Research explores different approaches to improve ML dependability by proposing new models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks. In this paper, we review and organize practical ML techniques that can improve the safety and dependability of ML algorithms and therefore ML-based software. Our organization maps state-of-the-art ML techniques to safety strategies in order to enhance the dependability of the ML algorithm from different aspects, and discuss research gaps as well as promising solutions.

show abstract

“…When we train an agent to achieve a goal, we usually want it to achieve that goal following implicit safety constraints. Handcrafting such safety constraints would be time-consuming, difficult to scale for complex problems, and might lead to reward hacking; so a reasonable proxy consists in limiting irreversible side-effects in the environment [26].…”

Section: Learning Reversible Policiesmentioning

confidence: 99%

There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning

Grinsztajn¹,

Ferret²,

Pietquin³

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose to learn to distinguish reversible from irreversible actions for better informed decision-making in Reinforcement Learning (RL). From theoretical considerations, we show that approximate reversibility can be learned through a simple surrogate task: ranking randomly sampled trajectory events in chronological order. Intuitively, pairs of events that are always observed in the same order are likely to be separated by an irreversible sequence of actions. Conveniently, learning the temporal order of events can be done in a fully self-supervised way, which we use to estimate the reversibility of actions from experience, without any priors. We propose two different strategies that incorporate reversibility in RL agents, one strategy for exploration (RAE) and one strategy for control (RAC). We demonstrate the potential of reversibility-aware agents in several environments, including the challenging Sokoban game. In synthetic tasks, we show that we can learn control policies that never fail and reduce to zero the side-effects of interactions, even without access to the reward function.* Equal contribution.

show abstract

AI Safety Gridworlds

Cited by 65 publications

References 0 publications

Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)

Scalar reward is not enough: A response to Silver, Singh, Precup and Sutton (2021)

Taxonomy of Machine Learning Safety: A Survey and Primer

There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning

Contact Info

Product

Resources

About