Safety Assessment for Autonomous Systems' Perception Capabilities

Abstract

Autonomous Systems (AS) are increasingly proposed, or used, in Safety Critical (SC) applications. Many such systems make use of sophisticated sensor suites and processing to provide scene understanding which informs the AS' decision-making. The sensor processing typically makes use of Machine Learning (ML) and has to work in challenging environments, further the ML-algorithms have known limitations,e.g., the possibility of false-negatives or false-positives in object classification. The well-established safety-analysis methods developed for conventional SC systems are not well-matched to AS, ML, or the sensing systems used by AS. This paper proposes an adaptation of well-established safety-analysis methods to address the specifics of perception-systems for AS, including addressing environmental effects and the potential failure-modes of ML, and provides a rationale for choosing particular sets of guidewords, or prompts, for safety-analysis. It goes on to show how the results of the analysis can be used to inform the design and verification of the AS and illustrates the new method by presenting a partial analysis of a road vehicle. Illustrations in the paper are primarily based on optical sensing, however the paper discusses the applicability of the method to other sensing modalities and its role in a wider safety process addressing the overall capabilities of AS.

The Safety of Autonomy: A Systematic Approach

“…Two other processes "bridge" between SACE and AMLAS: › Safety assurance of understanding in ASs (SAUS) 13 : conducting a safety analysis of the sense and understand elements of the system, seeking to identify how errors could arise in the world model that could lead to hazardous decision making › Safety assurance of decision making in ASs (SADA) 14 : conducting a safety analysis of the decision-making (and actuation) elements of the system, seeking to identify decisions that would be undesirable, even given a "perfect" world model.…”

A Risk-Based Decision-Making Process for Autonomous Trains Using POMDP: Case of the Anti-Collision Function

“…Addressing uncertainties in decision-making for autonomous systems had emerged as central research focus, identifying key problematics such as sensor fusion [27,28], perception under varying environmental conditions [29,30], and dynamic system state evaluation [31,32]. These challenges are critical as they directly impact the safety and reliability of autonomous operations.…”