Online discovery of AUV control policies to overcome thruster failures

Abstract

Abstract-We investigate methods to improve fault-tolerance of Autonomous Underwater Vehicles (AUVs) to increase their reliability and persistent autonomy. We propose a learningbased approach that is able to discover new control policies to overcome thruster failures as they happen. The proposed approach is a model-based direct policy search that learns on an on-board simulated model of the AUV. The model is adapted to a new condition when a fault is detected and isolated. Since the approach generates an optimal trajectory, the learned fault-tolerant policy is able to navigate the AUV towards a specified target with minimum cost. Finally, the learned policy is executed on the real robot in a closed-loop using the state feedback of the AUV. Unlike most existing methods which rely on the redundancy of thrusters, our approach is also applicable when the AUV becomes under-actuated in the presence of a fault. To validate the feasibility and efficiency of the presented approach, we evaluate it with three learning algorithms and three policy representations with increasing complexity. The proposed method is tested on a real AUV, Girona500.

Multi-objective reinforcement learning for AUV thruster failure recovery

“…However, the other two thrusters are considered fully functional (α 1 = α 3 = 1). Unlike our previous work [4], our proposed approach benefits from the remaining functionality of the faulty thruster together with the other healthy thrusters. For each experiment we specify a final time, T , for each episode (e.g.…”

“…In future, we plan to evaluate the efficiency of the proposed framework using real-world experiments, similar to our experiments with scalarized objectives in [4]. Therefore, the idea behind this experiment is to benefit from the previous experience and increase the performance of the learning approach by decreasing the computation time.…”

“…The backbone of the algorithm is based on weighted difference between solutions to perturb the population and to create candidate solutions. Previously, we compared the performance of single-objective DE with two other population based algorithms [4][5][6]. In order to solve the described multi-objective problem in this paper, a multi-objective differential evolution algorithm is utilized.…”

“…The approach results in expected rewards lying in a particular region of reward space. An alternative approach to specifying preferences is linear scalarization, [4], [17], [18], in which the objective vector is scalarized according to a weight vector. Varying the weights allows the user to express the relative importance of the objectives.…”

“…The proposed approach learns on an onboard simulated model of the AUV. In our previous research [4][5][6] fault-tolerant control policies have been discovered considering the assumption that the failure makes the thruster totally broken, meaning that a faulty thruster is equivalent to a thruster which is turned off. One of the contributions of this work is taking advantage of the remaining functionality of a partially broken thruster.…”