Autonomous Soaring Using Reinforcement Learning for Trajectory Generation

Woodbury, Timothy D.; Dunn, Caroline; Valasek, John

doi:10.2514/6.2014-0990

Cited by 30 publications

(16 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…13 considered the learning problem associated with finding the center of a stationary thermal without turbulence, and used a neural-based algorithm to recover the empirical rules proposed by Reichmann (14) to locate the core of the thermal. Other attempts (15,16) have used neural networks and Q-learning to find strategies to center a turbulence-free thermal. Akos et al (17) show that these simple rules fail even in the presence of modest velocity fluctuations modeled as Gaussian white noise, and express the need for strategies that could work in realistic turbulent flows.…”

mentioning

confidence: 99%

Learning to soar in turbulent environments

Reddy

Celani

Sejnowski

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

148

115

View full text Add to dashboard Cite

Birds and gliders exploit warm, rising atmospheric currents (thermals) to reach heights comparable to low-lying clouds with a reduced expenditure of energy. This strategy of flight (thermal soaring) is frequently used by migratory birds. Soaring provides a remarkable instance of complex decision making in biology and requires a longterm strategy to effectively use the ascending thermals. Furthermore, the problem is technologically relevant to extend the flying range of autonomous gliders. Thermal soaring is commonly observed in the atmospheric convective boundary layer on warm, sunny days. The formation of thermals unavoidably generates strong turbulent fluctuations, which constitute an essential element of soaring. Here, we approach soaring flight as a problem of learning to navigate complex, highly fluctuating turbulent environments. We simulate the atmospheric boundary layer by numerical models of turbulent convective flow and combine them with model-free, experience-based, reinforcement learning algorithms to train the gliders. For the learned policies in the regimes of moderate and strong turbulence levels, the glider adopts an increasingly conservative policy as turbulence levels increase, quantifying the degree of risk affordable in turbulent environments. Reinforcement learning uncovers those sensorimotor cues that permit effective control over soaring in turbulent environments.thermal soaring | turbulence | navigation | reinforcement learning M igrating birds and gliders use upward wind currents in the atmosphere to gain height while minimizing the energy cost of propulsion by the flapping of the wings or engines (1, 2). This mode of flight, called soaring, has been observed in a variety of birds. For instance, birds of prey use soaring to maintain an elevated vantage point in their search for food (3); migrating storks exploit soaring to cover large distances in their quest for greener pastures (4). Different forms of soaring have been observed. Of particular interest here is thermal soaring, where a bird gains height by using warm air currents (thermals) formed in the atmospheric boundary layer. For both birds and gliders, a crucial part of the thermal soaring is to identify a thermal and to find and maintain its core, where the lift is typically the largest. Once migratory birds have climbed up to the top of a thermal, they glide down to the next thermal and repeat the process, a migration strategy that strongly reduces energy costs (4). Soaring strategies are also important for technological applications, namely, the development of autonomous gliders that can fly large distances with minimal energy consumption (5).Thermals arise as ascending convective plumes driven by the temperature gradient created due to the heating of the earth's surface by the sun (6). Hydrodynamic instabilities and processes that lead to the formation of a thermal inevitably give rise to a turbulent environment characterized by strong, erratic fluctuations (7,8). Birds or gliders attempting to find and maintain a thermal face...

show abstract

mentioning

confidence: 99%

Learning to soar in turbulent environments

Reddy

Celani

Sejnowski

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

148

115

View full text Add to dashboard Cite

show abstract

“…Gudmondsson et al developed a lift seeking sink avoidance algorithm based on a potential flow method and best path search method and simulated flight through mountainous terrain [19]. Reinforcement learning strategies have also been applied to enable a UAV to autonomously harvest energy from the atmosphere [25][26][27] Since this area of work is relatively new, most of the literature simulates soaring methods without flight-testing. However, there are several notable exceptions.…”

Section: Acknowledgmentsmentioning

confidence: 99%

Artificial Lumbered Flight for Autonomous Soaring

Powers

Silverberg

Gopalarathnam

2020

Journal of Guidance, Control, and Dynamics

View full text Add to dashboard Cite

POWERS, THOMAS CORNELIUS. Artificial Lumbered Flight for Autonomous Soaring. (Under the direction of Drs. Larry Silverberg and Ashok Gopalarathnam).Soaring strategies are redefining the flight capabilities of small-class fixed-wing UAVs. This dissertation presents an autonomous soaring strategy that exploits updraft energy independent of the classification of an updraft. The strategy employs an artificial lumbered flight algorithm (ALFA) that weighs near-field updraft velocity estimates and mission priorities for navigation. This work raises the question of ALFA's ability to handle classified updrafts. Indeed, ALFA does not explicitly consider the classification of the updraft. Instead, ALFA measures updraft data along an aircraft's flight path, estimates updraft data ahead of the aircraft, generates candidate flightpaths ahead of the aircraft for evaluation, and then selects the best candidate flightpath based on a reward function. This dissertation describes the structure of ALFA and the tuning processes for the updraft estimator and the decision function.Flight results demonstrate the ability of artificial lumbered flight to harness atmospheric energy and complete its objectives. The flight results consider aircraft behavior in more detail, examining ALFA's effectiveness when flying among classified updrafts. They demonstrate the ability of artificial lumbered flight to navigate unclassified updrafts and harvest energy from thermal updrafts. Finally, this work highlights that autonomous flight design and control of small-class aircraft is maturing into its own flight regime that lies between the flapping flight and cruise flight regimes, and will be driven by the harvesting of energy from the atmosphere.

show abstract

“…A subfield of intelligent control, neurocontrol is characterized by the use of neural networks in control systems. Specifically, the use of reinforcement learning (RL) algorithms and deep neural networks has already been initially explored in the context of dynamic soaring [12][13][14][15] to train neural networks capable of exhibiting generalized and adaptable soaring behavior. That being said, the majority of the existing literature around the use of neural networks in aerial control systems has been focused on fixed-topology networks, where the structure of the nodes and connections are kept constant [16][17][18].…”

Section: Introductionmentioning

confidence: 99%

Neuroevolutionary Control for Autonomous Soaring

Kim

Perez

2021

Aerospace

View full text Add to dashboard Cite

The energy efficiency and flight endurance of small unmanned aerial vehicles (SUAVs) can be improved through the implementation of autonomous soaring strategies. Biologically inspired flight techniques such as dynamic and thermal soaring offer significant energy savings through the exploitation of naturally occurring wind phenomena for thrustless flight. Recent interest in the application of artificial intelligence algorithms for autonomous soaring has been motivated by the pursuit of instilling generalized behavior in control systems, centered around the use of neural networks. However, the topology of such networks is usually predetermined, restricting the search space of potential solutions, while often resulting in complex neural networks that can pose implementation challenges for the limited hardware onboard small-scale autonomous vehicles. In exploring a novel method of generating neurocontrollers, this paper presents a neural network-based soaring strategy to extend flight times and advance the potential operational capability of SUAVs. In this study, the Neuroevolution of Augmenting Topologies (NEAT) algorithm is used to train efficient and effective neurocontrollers that can control a simulated aircraft along sustained dynamic and thermal soaring trajectories. The proposed approach evolves interpretable neural networks in a way that preserves simplicity while maximizing performance without requiring extensive training datasets. As a result, the combined trajectory planning and aircraft control strategy is suitable for real-time implementation on SUAV platforms.

show abstract

Autonomous Soaring Using Reinforcement Learning for Trajectory Generation

Cited by 30 publications

References 8 publications

Learning to soar in turbulent environments

Learning to soar in turbulent environments

Artificial Lumbered Flight for Autonomous Soaring

Neuroevolutionary Control for Autonomous Soaring

Contact Info

Product

Resources

About