Deep reinforcement learning for conservation decisions

Abstract

1. Can machine learning help us make better decisions about a changing planet?In this paper, we illustrate and discuss the potential of a promising corner of machine learning known as deep reinforcement learning (RL) to help tackle the most challenging conservation decision problems. We provide a conceptual and technical introduction to deep RL as well as annotated code so that researchers can adopt, evaluate and extend these approaches.2. RL explicitly focuses on designing an agent who interacts with an environment that is dynamic and uncertain. Deep RL is the subfield of RL that incorporates deep neural networks into the agent. We train deep RL agents to solve sequential decision-making problems in setting fisheries quotas and managing ecological tipping points.3. We show that a deep RL agent is able to learn a nearly optimal solution for the fisheries management problem. For the tipping point problem, we show that a deep RL agent can outperform a sensible rule-of-thumb strategy. 4. Our results demonstrate that deep RL has the potential to solve challenging decision problems in conservation. While this potential may be compelling, the challenges involved in successfully deploying RL-based management to realistic scenarios are formidable-the required expertise and computational cost may place these applications beyond the reach of all but large, international technology firms. Ecologists must establish a better understanding of how these algorithms work and fail if we are to realize this potential and avoid the pitfalls such a transition would bring. We ultimately set forth a research framework based on well-posed, public challenges so that ecologists and computer scientists can collaborate towards solving hard decision-making problems in conservation.

Bridging adaptive management and reinforcement learning for more robust decisions

“…The problem set-up for RL closely mirrors model-based adaptive management (figure 1). Like classical model-based optimization approaches [16], an RL agent aims to learn a ‘policy’ (decision strategy) that maps the state of the system to the best action to take to maximize the expected sum of future rewards [20]. However, the process by which RL learns optimal policies can be fundamentally different.…”

“…But just because this may be possible, is it really a good idea? Adapting RL for adaptive management could open possibilities [20,76] but also introduces new pitfalls while re-surfacing age-old concerns of algorithmic decision processes [81]. We divide these possibilities and perils into three themes.…”

“…But unlike a human manager, RL algorithms can process near real-time and high-dimensional data as well as learn strategies by interacting with a wide diversity of simulations and scenarios [19] that would be infeasible for a human to process. Moreover, RL algorithms show promise to more effectively manage systems with unknown tipping points than do fixed 'rule-of-thumb' strategies [20].…”

Integrating machine learning, remote sensing and citizen science to create an early warning system for biodiversity

“…To tackle these issues, we have recently adapted an approach based on reinforcement learning, a branch of artificial intelligence that has been used in gaming and finances for several years, but had not yet been applied in conservation, even though similar goals have been pursued under various methodologies (Bone & Dragićević, 2010; Chadès et al, 2008, 2014; Lapeyrolerie et al, 2021; Marescot et al, 2013). We call our approach CAPTAIN: Conservation Area Prioritisation Through Artificial INtelligence (Silvestro et al, 2022).…”

Navigation between initial and desired community states using shortcuts

“…The problem has recently been conceptually explored in models (Angulo et al, 2019;Baranwal et al, 2022;Brias & Munch, 2021;Jones et al, 2020). Applications exist for example, fisheries, forestry, agriculture, and other natural resource management challenges where continual intervention is of interest (Boettiger et al, 2015;Krausman et al, 2013;Lapeyrolerie et al, 2022;Palmer et al, 2016), and also in microbial systems where metabolite production or infectious disease are priorities (Angulo et al, 2019;Costello et al, 2012;García-Jiménez et al, 2018). However, continuous control of multiple species' abundances becomes mathematically prohibitive and biologically unrealistic in high-richness communities.…”