Improving System Reliability Assessment of Safety-Critical Systems using Machine Learning Optimization Techniques

Alagoz, Ibrahim; Hoiss, Thomas; German, Reinhard

doi:10.25046/aj030107

Cited by 2 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Once a test session finishes, the test case effectiveness [54], i.e. whether it passes or fails, and the failure frequency [1], i.e. ratio of fail verdicts, can be measured from the test case report.…”

Section: Testing Informationmentioning

confidence: 99%

“…The inputs that the test case passes to the system and the outputs received are used to cluster similar test cases prior to prioritisation [61]. Also, the name and number of functions associated to system tests (system function) [1] have been applied to TCP to estimate reliability of a safety-critical system, whereas the similarity between usage patterns [6] allows including the impact of faults on different users in the TCP process. Some program-related attributes (test outputs and usage patterns) are expected to evolve as the SUT does.…”

Section: Relational Informationmentioning

confidence: 99%

“…Several authors explain that they opt for information attributes related to test case historical results and other properties (execution time, age, etc.) because access to the SUT code is not available in any form [1,72], or the company does not trace the relationship between test cases and code artefacts [36]. For those cases in which the company supports some kind of code instrumentation, the analysis of binary artefacts becomes a lighter process [103].…”

Section: Observed Limitations Of Tcp In Industrymentioning

confidence: 99%

“…Having a complex SUT also implies that its test suite will greatly differ from those available in public repositories. The industrial case studies analysed here report test suites with thousands of test cases [3,16,103], that takes several days or weeks to run [1,47,63], and that do not always correspond to unit testing [1,104,109]. In this sense, the cost of obtaining attributes with arity N:N will grow exponentially with the number of test cases, and its applicability will be ultimately conditioned by the involved artefacts and the efficiency of the extraction process.…”

Section: Observed Limitations Of Tcp In Industrymentioning

confidence: 99%

“…In the context of TCP, data quality is intrinsically linked to the identification of relevant information sources. Data acquisition and preparation become even more relevant when ML-based TCP methods are conceived for industrial application, as discussed in some recent case studies [1,16,94]. In particular, the heterogeneity and scale of the systems, the presence of networking functionalities and hardware-dependent test cases, or the need of parsing execution traces illustrate this point.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Ramírez,

Feldt,

Romero

2022

Preprint

View full text Add to dashboard Cite

Most software companies have extensive test suites and re-run parts of them continuously to ensure recent changes have no adverse effects. Since test suites are costly to execute, industry needs methods for test case prioritisation (TCP). Recently, TCP methods use machine learning (ML) to exploit the information known about the system under test (SUT) and its test cases. However, the value added by ML-based TCP methods should be critically assessed with respect to the cost of collecting the information. This paper analyses two decades of TCP research, and presents a taxonomy of 91 information attributes that have been used. The attributes are classified with respect to their information sources and the characteristics of their extraction process. Based on this taxonomy, TCP methods validated with industrial data and those applying ML are analysed in terms of information availability, attribute combination and definition of data features suitable for ML. Relying on a high number of information attributes, assuming easy access to SUT code and simplified testing environments are identified as factors that might hamper industrial applicability of ML-based TCP. The T PIA taxonomy provides a reference framework to unify terminology and evaluate alternatives considering the cost-benefit of the information attributes.CCS Concepts: • Software and its engineering → Software testing and debugging; • Computing methodologies → Machine learning.

show abstract

Section: Testing Informationmentioning

confidence: 99%

Section: Relational Informationmentioning

confidence: 99%

Section: Observed Limitations Of Tcp In Industrymentioning

confidence: 99%

Section: Observed Limitations Of Tcp In Industrymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Ramírez,

Feldt,

Romero

2022

Preprint

View full text Add to dashboard Cite

show abstract

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Ramírez

Feldt

Romero

2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

Most software companies have extensive test suites and re-run parts of them continuously to ensure recent changes have no adverse effects. Since test suites are costly to execute, industry needs methods for test case prioritisation (TCP). Recently, TCP methods use machine learning (ML) to exploit the information known about the system under test (SUT) and its test cases. However, the value added by ML-based TCP methods should be critically assessed with respect to the cost of collecting the information. This paper analyses two decades of TCP research, and presents a taxonomy of 91 information attributes that have been used. The attributes are classified with respect to their information sources and the characteristics of their extraction process. Based on this taxonomy, TCP methods validated with industrial data and those applying ML are analysed in terms of information availability, attribute combination and definition of data features suitable for ML. Relying on a high number of information attributes, assuming easy access to SUT code and simplified testing environments are identified as factors that might hamper industrial applicability of ML-based TCP. The TePIA taxonomy provides a reference framework to unify terminology and evaluate alternatives considering the cost-benefit of the information attributes.

show abstract

Improving System Reliability Assessment of Safety-Critical Systems using Machine Learning Optimization Techniques

Cited by 2 publications

References 22 publications

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Contact Info

Product

Resources

About