Robustness testing of autonomy software

Hutchison, Casidhe; Zizyte, Milda; Lanigan, Patrick E.; Guttendorf, David; Wagner, Michael; Goues, Claire Le; Koopman, Philip

doi:10.1145/3183519.3183534

Cited by 50 publications

(27 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Robustness testing became popular mostly due to its application to operating systems (OS) by Koopman et al [5], Krop et al [22], Shelton et al [27] among others [28]- [30], but its application to many other different types of systems (which hold different specificities) has been the object of research in numerous domains. Robustness testing techniques can be found, in general, applied to communication systems [7]- [9], embedded systems [10]- [12], [31], middleware [13]- [15], [32], web applications [17], [33], COTS [34]- [36], autonomous and adaptive systems [16], [25], [26], [37], and SOAP web services [18]- [20], [24], [38].…”

Section: Related Workmentioning

confidence: 99%

A Black Box Tool for Robustness Testing of REST Services

2021

View full text Add to dashboard Cite

REST services are nowadays being used to support many businesses, with most major companies exposing their services via REST interfaces (e.g., Google, Amazon, Instagram, and Slack). In this type of scenarios, heterogeneity is prevalent and software is sometimes exposed to unexpected conditions that may activate residual bugs, leading service operations to fail. Such failures may lead to financial or reputation losses (e.g., information disclosure). Although techniques and tools for assessing robustness have been thoroughly studied and applied to a large diversity of domains, REST services still lack practical approaches that specialize in robustness evaluation. In this paper, we present a tool (named bBOXRT) for performing robustness tests over REST services, solely based on minimal information expressed in their interface descriptions. We used bBOXRT to evaluate an heterogeneous set of 52 REST services that comprise 1,351 operations and fit in distinct categories (e.g., public, private, in-house). We were able to disclose several different types of robustness problems, including issues in services with strong reliability requirements and also a few security vulnerabilities. The results show that REST services are being deployed preserving software defects that harm service integration, and also carrying security vulnerabilities that can be exploited by malicious users.

show abstract

Section: Related Workmentioning

confidence: 99%

A Black Box Tool for Robustness Testing of REST Services

2021

View full text Add to dashboard Cite

show abstract

“…By using specifics of the reconfiguration model in [22] to generate the data to be injected, the work in [38] allows the tester to focus on specific parts of the sequence of configurations that enables adaptation policies to be tested. Robustness testing [18] describes a class of approaches that evaluate the degree to which a system or a component can function correctly in the presence of invalid inputs or in stressful environmental conditions. Our approach complements these works, as we mainly focus on establishing a test verdict and measuring a model coverage rather than generating test cases.…”

Section: Related Workmentioning

confidence: 99%

Testing adaptation policies for software components

2020

View full text Add to dashboard Cite

Self-adaptive systems have to implement adaptation policies described by sets of rules, that express how the components are reconfigured within the system, the priority of a given reconfiguration to happen, when a given (sequence of) event(s) occurs and when specific conditions on the system state are satisfied. However, when this priority is given by a fuzzy value (e.g. high, medium, low) depending on external and internal events, it has to be implemented inside the software with particular implementation choices made.This paper is dedicated to the validation of adaptation policies, using a modelbased testing approach, and a verdict establishment that is based on both the runtime verification of temporal properties, and the detection of inconsistencies between the adaptation policy and the reconfigurations implemented in the self-adaptive system. We propose a means to establish a test verdict based on the respect of the adaptation policy by the implementation, along with coverage measures of the rules. This provides an interesting feedback on the adaptation policy rules, allowing to detect reconfigurations that should not have occurred, high-priority reconfigurations that are never triggered, or low-priority reconfigurations that are too frequently executed, potential inconsistencies in the rules, or wrong interpretation of priorities. The test verdict is made based on the analysis of the execution traces of the system, which is stimulated using a usage model that describes the probabilities of external events to occur. An experiment, performed on a Vehicular Ad-hoc Network of autonomous vehicles, illustrates the interest of the approach. Context and MotivationsRecent years have seen the increase of cyber-physical systems which include a large scale of examples such as medical devices and systems, aerospace systems, transportation vehicles and intelligent highways with their associated problems such as security, safety and validity as mentionned in [28]. These systems are made of individual components

show abstract

“…An alternative to standard "photorealistic" simulation, which attempts to replace on-road testing, is to perform hardware-in-the-loop fault simulations, which stochastically introduce problems to the autonomous software at a much higher rate than might occur naturally. These approaches are similar to fault injection simulators (e.g., [55]) or software stress testing (e.g., [56]), and could reduce the burden of proof about the safety of autonomous systems to an attainable level.…”

Section: C) Holisticmentioning

confidence: 99%

A self driving license: Ensuring autonomous vehicles deliver on the promise of safer roads

Bradley

Preston

2020

MIT Science Policy Review

View full text Add to dashboard Cite

Upon maturation, autonomous vehicles (AVs) have the potential to provide significant benefit to society. A breadth of partially autonomous systems are already commercially available, and vehicles with advanced capabilities are tested and deployed on public roads. Although the advancement of AV technology is highly anticipated, the future of the industry currently rests on uncertain ground with respect to regulatory oversight. The current industry standards and legal regulations which apply to AVs are only equipped to fully ensure that simple autonomous capabilities are safe. As vehicles become more autonomous, and as driving decisions are shifted from human to computer, a regulatory paradigm shift will be necessary. This article reviews the present state of AV safety standards and regulations, and discusses the potential for regulatory evolution.

show abstract

Robustness testing of autonomy software

Cited by 50 publications

References 28 publications

A Black Box Tool for Robustness Testing of REST Services

A Black Box Tool for Robustness Testing of REST Services

Testing adaptation policies for software components

A self driving license: Ensuring autonomous vehicles deliver on the promise of safer roads

Contact Info

Product

Resources

About