Learning to soar in turbulent environments

Abstract

Birds and gliders exploit warm, rising atmospheric currents (thermals) to reach heights comparable to low-lying clouds with a reduced expenditure of energy. This strategy of flight (thermal soaring) is frequently used by migratory birds. Soaring provides a remarkable instance of complex decision making in biology and requires a longterm strategy to effectively use the ascending thermals. Furthermore, the problem is technologically relevant to extend the flying range of autonomous gliders. Thermal soaring is commonly observed in the atmospheric convective boundary layer on warm, sunny days. The formation of thermals unavoidably generates strong turbulent fluctuations, which constitute an essential element of soaring. Here, we approach soaring flight as a problem of learning to navigate complex, highly fluctuating turbulent environments. We simulate the atmospheric boundary layer by numerical models of turbulent convective flow and combine them with model-free, experience-based, reinforcement learning algorithms to train the gliders. For the learned policies in the regimes of moderate and strong turbulence levels, the glider adopts an increasingly conservative policy as turbulence levels increase, quantifying the degree of risk affordable in turbulent environments. Reinforcement learning uncovers those sensorimotor cues that permit effective control over soaring in turbulent environments.thermal soaring | turbulence | navigation | reinforcement learning M igrating birds and gliders use upward wind currents in the atmosphere to gain height while minimizing the energy cost of propulsion by the flapping of the wings or engines (1, 2). This mode of flight, called soaring, has been observed in a variety of birds. For instance, birds of prey use soaring to maintain an elevated vantage point in their search for food (3); migrating storks exploit soaring to cover large distances in their quest for greener pastures (4). Different forms of soaring have been observed. Of particular interest here is thermal soaring, where a bird gains height by using warm air currents (thermals) formed in the atmospheric boundary layer. For both birds and gliders, a crucial part of the thermal soaring is to identify a thermal and to find and maintain its core, where the lift is typically the largest. Once migratory birds have climbed up to the top of a thermal, they glide down to the next thermal and repeat the process, a migration strategy that strongly reduces energy costs (4). Soaring strategies are also important for technological applications, namely, the development of autonomous gliders that can fly large distances with minimal energy consumption (5).Thermals arise as ascending convective plumes driven by the temperature gradient created due to the heating of the earth's surface by the sun (6). Hydrodynamic instabilities and processes that lead to the formation of a thermal inevitably give rise to a turbulent environment characterized by strong, erratic fluctuations (7,8). Birds or gliders attempting to find and maintain a thermal face...

Efficient collective swimming by harnessing vortices through deep reinforcement learning

“…Reinforcement learning 29 has been introduced to identify navigation policies in several model systems of vortex dipoles, soaring birds and micro-swimmers. [30][31][32] Here, we expand on our earlier work 22, 33 combining Reinforcement Learning with Direct Numerical Simulations of the Navies stokes equations for two self-propelled and autonomous swimmers. We first investigate two-dimensional swimmers in a tandem configuration and analyse their kinematics for the cases of IS η and IS d (Fig.…”

Autonomous Thermalling as a Partially Observable Markov Decision Process

“…There is vast literature on automatically exploiting thermals [40,41,6,16,23,33,28], orographic lift [26,19], wind gusts [27], wind fields [31], and wind gradients [30,9], as well as on planning flight paths to extend fixed-wing sUAV endurance and range [17,31,10,11,28,14]. Soaring patterns have also been studied for birds [3].…”

“…In an early autonomous thermalling work, Wharington et al [40,41] used RL [37] in simulation to learn a thermalling strategy under several simplifying assumptions. Reddy et al [33] also used RL but removed some of Wharington and Herszberg [41]'s simplifications. In particular, they built a far more detailed thermal model that reflects updrafts' messy, turbulent nature.…”

“…Researchers have proposed several approaches to enable autonomous thermalling for fixed-wing sUAVs [40,41,6,16,23,33,28]. Generally, they rely on various parameterized thermal models characterizing vertical air velocity distribution within a thermal.…”

“…Reinforcement learning (RL) [37], a family of techniques for resolving such exploration-exploitation tradeoffs, has been considered in the context of autonomous thermalling, but only in simulation studies [40,41,33,28]. Its main practical drawback for this scenario is the episodic nature of classic RL algorithms.…”