A Safety Framework for Critical Systems Utilising Deep Neural Networks

Zhao, Xingyu; Banks, Alec; Sharp, James; Robu, Valentin; Flynn, David; Fisher, Michael; Huang, Xiaowei

doi:10.1007/978-3-030-54549-9_16

Cited by 37 publications

(20 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…-can be utilised to reason about high level assurance goals such as the system can operate correctly in the next 100 days with probability more than 99%. Research on this has started in [Zhao et al, 2020], where a Bayesian inference approach is taken.…”

Section: Learning-enabled Systemsmentioning

confidence: 99%

A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability

Huang

Kroening

Ruan

et al. 2020

Computer Science Review

Self Cite

289

142

View full text Add to dashboard Cite

Section: Learning-enabled Systemsmentioning

confidence: 99%

A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability

Huang

Kroening

Ruan

et al. 2020

Computer Science Review

Self Cite

289

142

View full text Add to dashboard Cite

“…CAE is often used as a framework in aviation, nuclear, and defense industries to reason about safety, security, reliability, and dependability. Recent work has begun applying CAE to the safety analysis of AI systems (Brundage et al, 2020;Zhao et al, 2020). We will adjust the concepts to apply to the seven requirements for trustworthy AI.…”

Section: The Resolve Phase: Verification Of Requirementsmentioning

confidence: 99%

On Assessing Trustworthy AI in Healthcare. Machine Learning as a Supportive Tool to Recognize Cardiac Arrest in Emergency Calls

et al. 2021

View full text Add to dashboard Cite

Artificial Intelligence (AI) has the potential to greatly improve the delivery of healthcare and other services that advance population health and wellbeing. However, the use of AI in healthcare also brings potential risks that may cause unintended harm. To guide future developments in AI, the High-Level Expert Group on AI set up by the European Commission (EC), recently published ethics guidelines for what it terms “trustworthy” AI. These guidelines are aimed at a variety of stakeholders, especially guiding practitioners toward more ethical and more robust applications of AI. In line with efforts of the EC, AI ethics scholarship focuses increasingly on converting abstract principles into actionable recommendations. However, the interpretation, relevance, and implementation of trustworthy AI depend on the domain and the context in which the AI system is used. The main contribution of this paper is to demonstrate how to use the general AI HLEG trustworthy AI guidelines in practice in the healthcare domain. To this end, we present a best practice of assessing the use of machine learning as a supportive tool to recognize cardiac arrest in emergency calls. The AI system under assessment is currently in use in the city of Copenhagen in Denmark. The assessment is accomplished by an independent team composed of philosophers, policy makers, social scientists, technical, legal, and medical experts. By leveraging an interdisciplinary team, we aim to expose the complex trade-offs and the necessity for such thorough human review when tackling socio-technical applications of AI in healthcare. For the assessment, we use a process to assess trustworthy AI, called 1Z-Inspection® to identify specific challenges and potential ethical trade-offs when we consider AI in practice.

show abstract

“…According to the survey, existing formal verification techniques for ML have been shown to work only in small-scale systems or by using approximation methods which provide an answer within a certain threshold. While mathematical approaches can, in some cases, be applied, a safety argument framework is required to address the partial knowledge and issues around the use of neural networks [66]. Clearly, the role for variations of testing is strong in machine learning, and a review of current and future activity in this area is given in [67].…”

Section: Verification and Validation Of Machine Learningmentioning

confidence: 99%

A Review of Verification and Validation for Space Autonomous Systems

et al. 2021

Self Cite

View full text Add to dashboard Cite

Purpose of Review The deployment of hardware (e.g., robots, satellites, etc.) to space is a costly and complex endeavor. It is of extreme importance that on-board systems are verified and validated through a variety of verification and validation techniques, especially in the case of autonomous systems. In this paper, we discuss a number of approaches from the literature that are relevant or directly applied to the verification and validation of systems in space, with an emphasis on autonomy. Recent Findings Despite advances in individual verification and validation techniques, there is still a lack of approaches that aim to combine different forms of verification in order to obtain system-wide verification of modular autonomous systems. Summary This systematic review of the literature includes the current advances in the latest approaches using formal methods for static verification (model checking and theorem proving) and runtime verification, the progress achieved so far in the verification of machine learning, an overview of the landscape in software testing, and the importance of performing compositional verification in modular systems. In particular, we focus on reporting the use of these techniques for the verification and validation of systems in space with an emphasis on autonomy, as well as more general techniques (such as in the aeronautical domain) that have been shown to have potential value in the verification and validation of autonomous systems in space.

show abstract

A Safety Framework for Critical Systems Utilising Deep Neural Networks

Cited by 37 publications

References 35 publications

A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability

A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability

On Assessing Trustworthy AI in Healthcare. Machine Learning as a Supportive Tool to Recognize Cardiac Arrest in Emergency Calls

A Review of Verification and Validation for Space Autonomous Systems

Contact Info

Product

Resources

About