Logic and Epistemology in Safety Cases

Rushby, John

doi:10.1007/978-3-642-40793-2_1

Cited by 16 publications

(13 citation statements)

References 11 publications

(11 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Section 3 we discussed that engineering a safe system always comes with a certain amount of risk [17] [18]. However, when we introduce a RL System the challenge of managing uncertainty (and thus risk) can significantly increase given that we are unable to ascertain, prior to deployment, that the RL algorithm will make a 'safe' decision.…”

Section: Risk Vs Benefitsmentioning

confidence: 99%

What Is Acceptably Safe for Reinforcement Learning?

Bragg

Habli

2018

Developments in Language Theory

View full text Add to dashboard Cite

Machine Learning algorithms are becoming more prevalent in critical systems where dynamic decision making and efficiency are the goal. As is the case for complex and safety-critical systems, where certain failures can lead to harm, we must proactively consider the safety assurance of such systems that use Machine Learning. In this paper we explore the implications of the use of Reinforcement Learning in particular, considering the potential benefits that it could bring to safety-critical systems, and our ability to provide assurances on the safety of systems incorporating such technology. We propose a high-level argument that could be used as the basis of a safety case for Reinforcement Learning systems, where the selection of 'reward' and 'cost' mechanisms would have a critical effect on the outcome of decisions made. We conclude with fundamental challenges that will need to be addressed to give the confidence necessary for deploying Reinforcement Learning within safetycritical applications.

show abstract

Section: Risk Vs Benefitsmentioning

confidence: 99%

What Is Acceptably Safe for Reinforcement Learning?

Bragg

Habli

2018

Developments in Language Theory

View full text Add to dashboard Cite

show abstract

“…To John Rushby, an assurance case is composed of two components: communication and reasoning [25]. He argues that given enough parameters, it is possible to strive for the "possibly perfect" piece of software [24].…”

Section: Quantitative: Pascalian Probabilitymentioning

confidence: 99%

Reasoning About Confidence and Uncertainty in Assurance Cases: A Survey

Duan

Rayadurgam

Heimdahl

et al. 2017

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Assurance cases are structured logical arguments supported by evidence that explain how systems, possibly software systems, satisfy desirable properties for safety, security or reliability. The confidence in both the logical reasoning and the underlying evidence is a factor that must be considered carefully when evaluating an assurance case; the developers must have confidence in their case before the system is delivered and the assurance case reviewer, such as a regulatory body, must have adequate confidence in the case before approving the system for use. A necessary aspect of gaining confidence in the assurance case is dealing with uncertainty, which may have several sources. Uncertainty, often impossible to eliminate, nevertheless undermines confidence and must therefore be sufficiently bounded. It can be broadly classified into two types, aleatory (statistical) and epistemic (systematic). This paper surveys how researchers have reasoned about uncertainty in assurance cases. We analyze existing literature to identify the type of uncertainty addressed and distinguish between qualitative and quantitative approaches for dealing with uncertainty.

show abstract

“…Rushby makes a distinction between two aspects of assurance arguments [20]: (1) logic doubt that relates to the reasoning in the argument and (2) epistemic doubt that relates to our understanding of the system and its environment. RushbyÕs thesis is that it is possible to use formalism, supported by automation, to eliminate logic doubt, i.e.…”

Section: Related Workmentioning

confidence: 99%

Weaving an Assurance Case from Design: A Model-Based Approach

Hawkins

Habli

Kolovos

et al. 2015

2015 IEEE 16th International Symposium on High Assurance Systems Engineering

View full text Add to dashboard Cite

AbstractÑ Assurance cases are used to demonstrate confidence in properties of interest for a system, e.g. for safety or security. A model-based assurance case seeks to bring the benefits of model-driven engineering, such as automation, transformation and validation, to what is currently a lengthy and informal process. In this paper we develop a model-based assurance approach, based on a weaving model, which allows integration between assurance case, design and process models and metamodels. In our approach, the assurance case itself is treated as a structured model, with the aim that all entities in the assurance case become linked explicitly to the models that represent them. We show how it is possible to exploit the weaving model for automated generation of assurance cases. Building upon these results, we discuss how a seamless modeldriven approach to assurance cases can be achieved and examine the utility of increased formality and automation.

show abstract

Logic and Epistemology in Safety Cases

Cited by 16 publications

References 11 publications

What Is Acceptably Safe for Reinforcement Learning?

What Is Acceptably Safe for Reinforcement Learning?

Reasoning About Confidence and Uncertainty in Assurance Cases: A Survey

Weaving an Assurance Case from Design: A Model-Based Approach

Contact Info

Product

Resources

About