QoS-aware Metamorphic Testing: An Elevation Case Study

Ayerdi, Jon; Segura, Sergio; Arrieta, Aitor; Sagardui, Goiuria; Arratibel, Maite

doi:10.1109/issre5003.2020.00019

Cited by 17 publications

(25 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In these cases, the system was not the dispatching algorithm but other software components of the elevators. In our previous paper, we used the technique metamorphic testing to test the dispatching algorithm of Orona, 76 showing promising results. A problem with metamorphic testing was the difficulty to extract effective metamorphic relations.…”

Section: Related Workmentioning

confidence: 99%

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

The software of systems of elevators needs constant maintenance to deal with new functionality, bug fixes, or legislation changes. To automatically validate the software of these systems, a typical approach in industry is to use regression oracles, which execute test inputs both in the software version under test and in a previous software version. However, these practices require a long test execution time and cannot be re‐used at different test phases. To deal with these issues, we propose Dispatching AlgoRIthm Oracle (DARIO), a test oracle that relies on regression machine‐learning algorithms to detect both functional and non‐functional problems of the system. The machine‐learning algorithms of this oracle are trained by using data from previously tested versions to predict reference functional and non‐functional performance values of the new versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach. A total of five regression learning algorithms were validated by using mutation testing techniques. For the context of functional bugs, the accuracy when predicting verdicts by DARIO ranged between 95% and 98%, across the different scenarios proposed. For the context of non‐functional bugs, were competitive too, having an accuracy when predicting verdicts by DARIO ranged between 83% and 87%.

show abstract

Section: Related Workmentioning

confidence: 99%

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

show abstract

“…The dispatching algorithm is a critical component in a system of elevators. Indeed, a wrong assignment of calls can have a huge impact on the overall Quality of Service (QoS) [5]. Orona has developed a large corpus of dispatching algorithms, making it necessary to employ cost-effective and automated testing solutions for their verification and validation.…”

Section: Industrial Case Studymentioning

confidence: 99%

“…For each test execution, Elevate returns the values of domain specific QoS metrics (e.g., AWT, energy consumption). In our study, we considered the three QoS metrics used in our previous work [5]:…”

Section: Industrial Case Studymentioning

confidence: 99%

“…Metamorphic Testing has been successfully applied in the domain of CPSs, e.g., for testing wireless sensor networks [13], autonomous drones [25], or self-driving cars [41,46]. Metamorphic testing was also investigated in the context of non-functional properties, such as performance or QoS metrics [5,11,21,35,36].…”

Section: Introductionmentioning

confidence: 99%

“…Recently, Ayerdi et al proposed an approach based on metamorphic testing that considered QoS metrics from the elevation domain at Orona, showing promising results [5]. However, the definition of effective MRs was possible only with in-depth knowledge of the domain and the system under test [15,32].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study

Ayerdi

Terragni

Arrieta

et al. 2021

Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

Self Cite

View full text Add to dashboard Cite

One of the major challenges in the verification of complex industrial Cyber-Physical Systems is the difficulty of determining whether a particular system output or behaviour is correct or not, the socalled test oracle problem. Metamorphic testing alleviates the oracle problem by reasoning on the relations that are expected to hold among multiple executions of the system under test, which are known as Metamorphic Relations (MRs). However, the development of effective MRs is often challenging and requires the involvement of domain experts. In this paper, we present a case study aiming at automating this process. To this end, we implemented GAssertMRs, a tool to automatically generate MRs with genetic programming. We assess the cost-effectiveness of this tool in the context of an industrial case study from the elevation domain. Our experimental results show that in most cases GAssertMRs outperforms the other baselines, including manually generated MRs developed with the help of domain experts. We then describe the lessons learned from our experiments and we outline the future work for the adoption of this technique by industrial practitioners. CCS CONCEPTS• Computer systems organization → Embedded software; • Theory of computation → Assertions; • Software and its engineering → Software testing and debugging; • Computing methodologies → Genetic programming.

show abstract

A microservice-based framework for multi-level testing of cyber-physical systems

et al. 2023

View full text Add to dashboard Cite

In the last years, the use of microservice architectures is spreading in Cyber-Physical Systems (CPSs) and Internet of Things (IoT) domains. CPSs are systems that integrate digital cyber computations with physical processes. The development of software for CPSs demands a constant maintenance to support new requirements, bug fixes, and deal with hardware obsolescence. The key in this process is code testing and more if the code is fragmented during the development of CPSs. It is important to remark that this process is challenging and time-consuming. In this paper, we report on the experience of instantiating of the microservice-based architecture for DevOps of CPSs to test elevator dispatching algorithms across different test levels (i.e., SiL, HiL and Operation). Such an architecture allows for a continuous deployment, monitoring and validation of CPSs. By integrating the approach with a real industrial case study, we demonstrate that our approach reduces significantly the time needed in the testing process and consequently, reduces the economic cost of the entire process.

show abstract

QoS-aware Metamorphic Testing: An Elevation Case Study

Cited by 17 publications

References 38 publications

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study

A microservice-based framework for multi-level testing of cyber-physical systems

Contact Info

Product

Resources

About