Mind the gaps: Assuring the safety of autonomous systems from an engineering, ethical, and legal perspective

Burton, Simon; Habli, Ibrahim; Lawton, Tom; McDermid, John; Morgan, Phillip; Porter, Zoë

doi:10.1016/j.artint.2019.103201

Cited by 101 publications

(70 citation statements)

References 26 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As we progress to Level 5, both the magnitude of disruption and opportunity will become greater. With this greater power, AI will have to assume greater responsibility, and it will be thus crucial to develop 'responsible and ethical AI' before we get to Level 5 [170][171][172]. From an urban point of view, AI technology is progressing fast, thereby gaining more and more autonomy in cities.…”

mentioning

confidence: 99%

The Sustainability of Artificial Intelligence: An Urbanistic Viewpoint from the Lens of Smart and Sustainable Cities

2020

View full text Add to dashboard Cite

The popularity and application of artificial intelligence (AI) are increasing rapidly all around the world—where, in simple terms, AI is a technology which mimics the behaviors commonly associated with human intelligence. Today, various AI applications are being used in areas ranging from marketing to banking and finance, from agriculture to healthcare and security, from space exploration to robotics and transport, and from chatbots to artificial creativity and manufacturing. More recently, AI applications have also started to become an integral part of many urban services. Urban artificial intelligences manage the transport systems of cities, run restaurants and shops where every day urbanity is expressed, repair urban infrastructure, and govern multiple urban domains such as traffic, air quality monitoring, garbage collection, and energy. In the age of uncertainty and complexity that is upon us, the increasing adoption of AI is expected to continue, and so its impact on the sustainability of our cities. This viewpoint explores and questions the sustainability of AI from the lens of smart and sustainable cities, and generates insights into emerging urban artificial intelligences and the potential symbiosis between AI and a smart and sustainable urbanism. In terms of methodology, this viewpoint deploys a thorough review of the current status of AI and smart and sustainable cities literature, research, developments, trends, and applications. In so doing, it contributes to existing academic debates in the fields of smart and sustainable cities and AI. In addition, by shedding light on the uptake of AI in cities, the viewpoint seeks to help urban policymakers, planners, and citizens make informed decisions about a sustainable adoption of AI.

show abstract

mentioning

confidence: 99%

The Sustainability of Artificial Intelligence: An Urbanistic Viewpoint from the Lens of Smart and Sustainable Cities

2020

View full text Add to dashboard Cite

show abstract

“…The worst-case 3-point joint prior distribution that gives the infimum for Eq. (4) when φ > 1 − θ, subject to constraints (2) and(5).…”

mentioning

confidence: 99%

Assessing safety-critical systems from operational testing: A study on autonomous vehicles

Zhao

Salako

Strigini

et al. 2020

Information and Software Technology

View full text Add to dashboard Cite

Context: Demonstrating high reliability and safety for safety-critical systems (SCSs) remains a hard problem. Diverse evidence needs to be combined in a rigorous way: in particular, results of operational testing with other evidence from design and verification. Growing use of machine learning in SCSs, by precluding most established methods for gaining assurance, makes evidence from operational testing even more important for supporting safety and reliability claims. Objective: We revisit the problem of using operational testing to demonstrate high reliability. We use Autonomous Vehicles (AVs) as a current example. AVs are making their debut on public roads: methods for assessing whether an AV is safe enough are urgently needed. We demonstrate how to answer 5 questions that would arise in assessing an AV type, starting with those proposed by a highly-cited study. Method: We apply new theorems extending our Conservative Bayesian Inference (CBI) approach, which exploit the rigour of Bayesian methods while reducing the risk of involuntary misuse associated (we argue) with now-common applications of Bayesian inference; we define additional conditions needed for applying these methods to AVs. Results: Prior knowledge can bring substantial advantages if the AV design allows strong expectations of safety before road testing. We also show how naive attempts at conservative assessment may lead to over-optimism instead; why extrapolating the trend of disengagements (take-overs by human drivers) is not suitable for safety claims; use of knowledge that an AV has moved to a "less stressful" environment. Conclusion: While some reliability targets will remain too high to be practically verifiable, our CBI approach removes a major source of doubt: it allows use of prior knowledge without inducing dangerously optimistic biases. For certain ranges of required reliability and prior beliefs, CBI thus supports feasible, sound arguments. Useful conservative claims can be derived from limited prior knowledge.

show abstract

“…Indeed, at the moment, there is no single definition of transparency in the literature ( Theodorou et al, 2017 ; Larsson and Heintz, 2020 ), as it varies depending on its application domain ( Weller, 2019 ) and its dimensions ( Bertino et al, 2019 ). The notion of transparency is also often interwoven with other related concepts such as fairness ( Olhede and Rodrigues, 2017 ), trustworthiness ( Wortham, 2020 ; Nesset et al, 2021 ), interpretability ( Gilpin et al, 2018 ), accountability ( Koene et al, 2019 ), dependability ( TaheriNejad et al, 2020 ), reliability ( Wright et al, 2020 ), and/or safety ( Burton et al, 2020 ).…”

Section: Related Workmentioning

confidence: 99%

IEEE P7001: A Proposed Standard on Transparency

et al. 2021

View full text Add to dashboard Cite

This paper describes IEEE P7001, a new draft standard on transparency of autonomous systems1. In the paper, we outline the development and structure of the draft standard. We present the rationale for transparency as a measurable, testable property. We outline five stakeholder groups: users, the general public and bystanders, safety certification agencies, incident/accident investigators and lawyers/expert witnesses, and explain the thinking behind the normative definitions of “levels” of transparency for each stakeholder group in P7001. The paper illustrates the application of P7001 through worked examples of both specification and assessment of fictional autonomous systems.

show abstract

Mind the gaps: Assuring the safety of autonomous systems from an engineering, ethical, and legal perspective

Cited by 101 publications

References 26 publications

The Sustainability of Artificial Intelligence: An Urbanistic Viewpoint from the Lens of Smart and Sustainable Cities

The Sustainability of Artificial Intelligence: An Urbanistic Viewpoint from the Lens of Smart and Sustainable Cities

Assessing safety-critical systems from operational testing: A study on autonomous vehicles

IEEE P7001: A Proposed Standard on Transparency

Contact Info

Product

Resources

About