Learning to school in dense configurations with multi-agent deep reinforcement learning

Abstract

Fish are observed to school in different configurations. However, how and why fish maintain a stable schooling formation still remains unclear. This work presents a numerical study of the dense schooling of two freely-swimming swimmers by a hybrid method of the multi-agent deep reinforcement learning and the immersed boundary-lattice Boltzmann method. Active control policies are developed by synchronously training the leader to swim at given speed and orientation and the follower to hold close proximity to the leader. After training, the swimmers could resist the strong hydrodynamic force to remain in stable formations and meantime swim in desired path, only by their tail-beat flapping. The tail movement of the swimmers in the stable formations are irregular and asymmetrical, indicating the swimmers are carefully adjusting their body-kinematics to balance the hydrodynamic force. In addition, a significant decrease in the mean amplitude and cost of transport is found for the follower in the latter two cases, indicating these swimmers could maintain the swimming speed with less efforts. The results also show that the side-by-side formation is hydrodynamically more stable but energetically less efficient than other configurations, while the full-body staggered formation is energetically more efficient as a whole.

Special issue: bioinspired fluid-structure interaction

“…Beyond the problem of flapping-based locomotion of an agent in a homogeneous environment, a few papers in this Special Issue examine the situation of moving in a more complex environment, such as the perturbed environment produced by a neighbor, or in intrinsically non-homogeneous fluid. Zhu et al [9] present a numerical study on the collective motion of two fish-like swimmers in fluids. They use a hybrid method that couples CFD and machine learning, where the two swimmers are trained to learn their control strategies using the deep reinforcement learning method.…”

Optimized hydrodynamic interactions in phalanx school arrays of accelerated thunniform swimmers

“…Formal optimization studies that aim to identify effective schooling configurations from a hydrodynamic perspective are absent in the literature, although works devoted to an optimization of a swimming performance of a solitary swimmer can be noted [14,23,26,27]. Some recent studies also considered the problem of control to ensure that a swimmer can successfully follow a leader on a specified trajectory [16,28,29]. The current work does not consider the effects of control, but rather is focused on identifying the hydrodynamically-optimum maintained configurations, which can serve as targets for control strategies.…”