Testing Web Enabled Simulation at Scale Using Metamorphic Testing

Ahlgren, John; Berezin, Maria Eugenia; Bojarczuk, Kinga; Dulskyte, Elena; Dvortsova, Inna; George, Johann; Gucevska, Natalija; Harman, Mark; Lomelí, María; Meijer, Erik; Sapora, Silvia; Spahr-Summers, Justin

doi:10.1109/icse-seip52600.2021.00023

Cited by 36 publications

(15 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Successful applications of MT have been reported in multiple domains including web services and applications, machine learning, compilers, cybersecurity, and bioinformatics, among others [19], [50]. Industrial applications of MT have been reported at Google [24] and Facebook [2].…”

Section: B Metamorphic Testingmentioning

confidence: 99%

“…4) Experimental Setup: The test cases are based on a template project from a real building with 10 floors and up to 6 elevators. For the MRs, our inputs are: (1) The set of available elevators (at least 2, and up to 6), including their positions (floors 1 through 10), (2) The set of relevant elevator parameters to change their speed (explained later on), and (3) The passengers list, where the arrival time, source floor, destination floor, and passenger weights are variable and the rest of the parameters are set to default values. The rest of the building parameters, elevator specs, etc.…”

Section: B Experiments 1: Elevator Control Systemmentioning

confidence: 99%

See 1 more Smart Citation

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

et al. 2023

View full text Add to dashboard Cite

Cyber-Physical Systems (CPSs) are a new generation of systems which integrate software with physical processes. The increasing complexity of these systems, combined with the uncertainty in their interactions with the physical world, makes the definition of effective test oracles especially challenging, facing the well known test oracle problem. Metamorphic testing has shown great potential to alleviate the test oracle problem by exploiting the relations among the inputs and outputs of different executions of the system, so-called metamorphic relations (MRs). In this article, we propose a MR pattern called Performance Variation (PV) for the identification of performance-driven MRs, and we show its applicability in two CPSs from different domains: automated navigation systems and elevator control systems. For the evaluation, we assessed the effectiveness of this approach for detecting failures in an open source simulation-based autonomous navigation system, as well as in an industrial case study from the elevation domain. We derive concrete MRs based on the PV pattern for both case studies and we evaluate their effectiveness with seeded faults. Results show that the approach is effective at detecting over 88% of the seeded faults, while keeping the ratio of false positives at 4% or lower.

show abstract

Section: B Metamorphic Testingmentioning

confidence: 99%

Section: B Experiments 1: Elevator Control Systemmentioning

confidence: 99%

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

et al. 2023

View full text Add to dashboard Cite

show abstract

“…The shift-left strategy has been widely adopted by Meta for various types of issues e.g., reliability, performance, privacy and security. It has been used in testing client-side [5] as well as server-side (as reported here), and also has been used to test Meta's simulation platform, WW [18]. Meta's approach to the shift left philosophy is not restricted to testing, but has also been used in static analysis [19].…”

Section: Fausta System Deploymentmentioning

confidence: 99%

FAUSTA: Scaling Dynamic Analysis with Traffic Generation at WhatsApp

Mao¹,

Kapus²,

Petrou³

et al. 2022

2022 IEEE Conference on Software Testing, Verification and Validation (ICST)

Self Cite

View full text Add to dashboard Cite

“…In this example, 𝑥 1 is a source (base) test, and 𝑥 2 is a follow-up constructed by transforming the source to satisfy the metamorphic relation between 𝑥 1 and 𝑥 2 and check the equality between outcomes. Metamorphic properties have been used to validate machine learning classifiers [31], autonomous vehicle [32], Google/Yahoo search engine [33], and social media apps like Facebook [4], but not used for tax preparation software.…”

Section: Introductionmentioning

confidence: 99%

Metamorphic Testing and Debugging of Tax Preparation Software

Tizpaz-Niari¹,

Wagner²,

Darian³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper presents a data-driven debugging framework to improve the trustworthiness of US tax preparation software. As the US tax law evolves to adapt to ever-changing politico-economic realities, tax preparation software plays a significant role in helping taxpayers navigate these complexities. Given the legal implications of bugs in such software on its users, ensuring the compliance and trustworthiness of tax preparation software is of paramount importance.The key obstacles in developing debugging aids for tax preparation systems, however, are the unavailability of explicit specifications and the difficulty of obtaining oracles.We posit that, since the US tax law adheres to the legal doctrine of precedent, the specifications about the outcome of tax preparation software for an individual taxpayer must be viewed in comparison with individuals that are deemed similar. Consequently, these specifications are naturally available as properties on the software requiring similar inputs provide similar outputs. Inspired by metamorphic testing framework from software engineering, we dub these relations metamorphic relations.We explicated metamorphic relations for a set of properties by carefully studying the US Form 1040 as implemented in an opensource tax preparation software. We developed a randomized search strategy to explore the space of metamorphic relations and explain the behaviors of software with a classification tree algorithm. Our tool revealed 4 types of failures in the software and provided intuitive explanations to understand the failures.

show abstract

Testing Web Enabled Simulation at Scale Using Metamorphic Testing

Cited by 36 publications

References 51 publications

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

Performance-Driven Metamorphic Testing of Cyber-Physical Systems

FAUSTA: Scaling Dynamic Analysis with Traffic Generation at WhatsApp

Metamorphic Testing and Debugging of Tax Preparation Software

Contact Info

Product

Resources

About