Efficient collective swimming by harnessing vortices through deep reinforcement learning

Abstract

Fish in schooling formations navigate complex flow fields replete with mechanical energy in the vortex wakes of their companions. Their schooling behavior has been associated with evolutionary advantages including energy savings, yet the underlying physical mechanisms remain unknown. We show that fish can improve their sustained propulsive efficiency by placing themselves in appropriate locations in the wake of other swimmers and intercepting judiciously their shed vortices. This swimming strategy leads to collective energy savings and is revealed through a combination of high-fidelity flow simulations with a deep reinforcement learning (RL) algorithm. The RL algorithm relies on a policy defined by deep, recurrent neural nets, with long-short-term memory cells, that are essential for capturing the unsteadiness of the two-way interactions between the fish and the vortical flow field. Surprisingly, we find that swimming in-line with a leader is not associated with energetic benefits for the follower. Instead, "smart swimmer(s)" place themselves at off-center positions, with respect to the axis of the leader(s) and deform their body to synchronize with the momentum of the oncoming vortices, thus enhancing their swimming efficiency at no cost to the leader(s). The results confirm that fish may harvest energy deposited in vortices and support the conjecture that swimming in formation is energetically advantageous. Moreover, this study demonstrates that deep RL can produce navigation algorithms for complex unsteady and vortical flow fields, with promising implications for energy savings in autonomous robotic swarms.

Machine Learning for Fluid Mechanics

“…It is important to emphasize that RL requires significant computational resources due to the large numbers of episodes required to properly account for the interaction of the agent and the environment. This cost may be trivial for games but it may be prohibitive in experiments and flow simulations, a situation that is rapidly changing (Verma et al 2018).…”

On the energetics and stability of a minimal fish school

“…309 However, one should be aware that our conclusion is based on the tethered motion 310 (fixed CoM) and absence of kinematic adjustment. A recent study by Verma et al [27] 311 shows that, when learning-based optimized kinematic adjustment is present, wake 312 capture can be advantegeous. Therefore, the comparison between the present study and 313 study of Verma et al [27] demonstrates that there exists a distinction between wake 314 capturing and wake energy harvesting: successful wake capture requires skills in sensing 315 and adjustment, and if the fish (or an artificial swimmer) lacks those skills, wake 316 capture may become energetically unfavorable.…”

“…A recent study by Verma et al [27] 311 shows that, when learning-based optimized kinematic adjustment is present, wake 312 capture can be advantegeous. Therefore, the comparison between the present study and 313 study of Verma et al [27] demonstrates that there exists a distinction between wake 314 capturing and wake energy harvesting: successful wake capture requires skills in sensing 315 and adjustment, and if the fish (or an artificial swimmer) lacks those skills, wake 316 capture may become energetically unfavorable. Besides the active mechanism, passive 317 mechanisms based on appropriate body flexibility and mass distribution are also 318 potential factors that may influence fish performance in school [25].…”

“…36% decrement of CoT in 2D and only 5% decrement in 3D, in Verma et 326 al. [27]). Further study is needed to quantify and fundamentally explain the difference 327 between hydrodynamic interactions in the two-dimensional and the three-dimensional 328 contexts.…”

Control of chaotic systems by deep reinforcement learning

“…Regardless of the optimized critic NN, the actor NN is also optimized according to (17). The optimality of the control is guaranteed by the Bellman principle under the hypothesis that the state-action space is known.…”