Efficient Navigation of Colloidal Robots in an Unknown Environment via Deep Reinforcement Learning

Abstract

Equipping micro-/nanoscale colloidal robots with artificial intelligence (AI) such that they can efficiently navigate in unknown complex environments can dramatically impact their use in emerging applications such as precision surgery and targeted nanodrug delivery. Herein, a model-free deep reinforcement learning algorithm is developed that trains colloidal robots to efficiently navigate in unknown environments with random obstacles. A deep neural network architecture is used that enables the colloidal robots to mimic animal navigation decision-making by directly processing raw sensor input and decomposing long-range navigations to short-range ones. The trained robot agents learn to make navigation decisions regarding both obstacle avoidance and travel time minimization, based solely on local sensory inputs without prior knowledge of the global environment. Such agents with biologically inspired mechanisms can acquire competitive navigation capabilities in large-scale, complex environments containing obstacles of diverse shapes, sizes, and configurations. Herein, the potential of AI to enable colloidal robots in extensive applications is illustrated.

Hydrodynamics can determine the optimal route for microswimmer navigation

“…Our results might be relevant for future studies on microswimmers in various complex environments involving hard walls or obstacle landscapes [65][66][67] , penetrable boundaries 68,69 , or external (viscosity) gradients [70][71][72] . For such scenarios, our results (or generalizations based on the same framework) can be used as reference calculations, e.g., to test machine-learning-based approaches to optimal microswimmer navigation 5,6 and perhaps also to help programming navigation systems for future microswimmer generations. They should also serve as a useful ingredient for future works on microswimmer navigation problems in environments which are not globally known but subsequently discovered by the microswimmers.…”

“…Likewise, navigation and search strategies are frequently encountered in a plethora of biological systems, including the foraging of animals for food 2 , or of T cells searching for targets to mount an immune response 3 . Very recently there is a growing interest also in optimal navigation problems and search strategies [4][5][6][7][8][9] of microswimmers [10][11][12][13] and "dry" active Brownian particles [14][15][16][17][18] . These active agents can self-propel in a low-Reynoldsnumber solvent, and might play a key role in tomorrow's nanomedicine as recently popularized, e.g.…”

“…Meanwhile, refs. 5,6,37 have used (deep) reinforcement learning to explore microswimmer navigation problems in mazes and obstacle arrays assuming global 5 or only local 6 knowledge of the environment. Very recent analytical approaches 7,8 to optimal active particle navigation complement these works and allow testing machine-learned results 8,9 .…”

The control effort to steer self-propelled microswimmers depends on their morphology: comparing symmetric spherical versus asymmetric L -shaped particles

“…However, the direction is still governed by diffusion and therefore the particle trajectory is still random. To overcome this, external control strategies have been proposed allowing microswimmers to be kept in place or to navigate them to specific targets [5,6,11,12,[14][15][16][17][18][19]. This is achieved by acquiring and processing information on the current state of the particle, e.g.…”

Reactive navigation under a fuzzy rules-based scheme and reinforcement learning for mobile robots

“…When the decision process implies reaching a predefined goal with obstacle avoidance capability, fuzzy logic or bioinspired methodologies can be used for a robot to decide between avoiding an obstacle, following a wall to the right, or to the left ( Zapata-Cortes, Acosta-Amaya & Jimnez-Builes, 2020 ; Khedher., Mziou. & Hadji., 2021 ; Yang, Bevan & Li, 2020 ; Wang, Hu & Ma, 2020 ). Within the decision-making process, it is possible to increase precision in parameters such as the speed, or the robot’s angle rotation, improving the executed movements in narrow paths ( Teja S & Alami, 2020 ).…”