On the use of usage patterns from telemetry data for test case prioritization

Anderson, Jeff; Azizi, Maral; Salem, Saeed; Do, Hyunsook

doi:10.1016/j.infsof.2019.05.008

Cited by 14 publications

(7 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The characteristics of the SUT impose some adaptations of TCP and its input information attributes. For instance, a web application with a wide catalogue of functionalities and types of users has benefited from the analysis of usage patterns to locate fault-prone parts [6]. This work was responsible for adding usage pattern as attribute in the taxonomy, as it was not considered in any non-industrial paper before.…”

Section: Observed Limitations Of Tcp In Industrymentioning

confidence: 99%

“…The inputs that the test case passes to the system and the outputs received are used to cluster similar test cases prior to prioritisation [61]. Also, the name and number of functions associated to system tests (system function) [1] have been applied to TCP to estimate reliability of a safety-critical system, whereas the similarity between usage patterns [6] allows including the impact of faults on different users in the TCP process. Some program-related attributes (test outputs and usage patterns) are expected to evolve as the SUT does.…”

Section: Relational Informationmentioning

confidence: 99%

“…Under controlled environments, test cases are unitary, independent and do not need special considerations to be executed. These assumptions do not hold in industrial case System testing level Rely on information used in manual testing [58] Test case dependencies Include dependency analysis [41,90,91] SUT Faults affect different users Definition of usage patterns as [6] charact. or functionalities attributes Heterogeneity of artefacts and languages Combination of attributes and ad-hoc preprocessing [16] Real-time conditions and specific standards Information at runtime during testing [109] studies, as reflected by the appearance of attributes for inclusion in the taxonomy that were specifically conceived to introduce practicalities.…”

Section: Observed Limitations Of Tcp In Industrymentioning

confidence: 99%

See 2 more Smart Citations

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Ramírez,

Feldt,

Romero

2022

Preprint

View full text Add to dashboard Cite

Most software companies have extensive test suites and re-run parts of them continuously to ensure recent changes have no adverse effects. Since test suites are costly to execute, industry needs methods for test case prioritisation (TCP). Recently, TCP methods use machine learning (ML) to exploit the information known about the system under test (SUT) and its test cases. However, the value added by ML-based TCP methods should be critically assessed with respect to the cost of collecting the information. This paper analyses two decades of TCP research, and presents a taxonomy of 91 information attributes that have been used. The attributes are classified with respect to their information sources and the characteristics of their extraction process. Based on this taxonomy, TCP methods validated with industrial data and those applying ML are analysed in terms of information availability, attribute combination and definition of data features suitable for ML. Relying on a high number of information attributes, assuming easy access to SUT code and simplified testing environments are identified as factors that might hamper industrial applicability of ML-based TCP. The T PIA taxonomy provides a reference framework to unify terminology and evaluate alternatives considering the cost-benefit of the information attributes.CCS Concepts: • Software and its engineering → Software testing and debugging; • Computing methodologies → Machine learning.

show abstract

Section: Observed Limitations Of Tcp In Industrymentioning

confidence: 99%

Section: Relational Informationmentioning

confidence: 99%

Section: Observed Limitations Of Tcp In Industrymentioning

confidence: 99%

See 1 more Smart Citation

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Ramírez,

Feldt,

Romero

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Product usage data helps to prioritize test cases and to synthesize realistic test data in the continuous verification step, see Step #2 in Fig. 2, (Anderson et al 2019).…”

Section: Continuous Monitoringmentioning

confidence: 99%

Towards cost-benefit evaluation for continuous software engineering activities

et al. 2022

View full text Add to dashboard Cite

Context: Software companies must become better at delivering software to remain relevant in the market. Continuous integration and delivery practices promise to streamline software deliveries to end-users by implementing an automated software development and delivery pipeline. However, implementing or retrofitting an organization with such a pipeline is a substantial investment, while the reporting on benefits and their relevance in specific contexts/domains are vague. Aim: In this study, we explore continuous software engineering practices from an investment-benefit perspective. We identify what benefits can be attained by adopting continuous practices, what the associated investments and risks are, and analyze what parameters determine their relevance. Method: We perform a multiple case study to understand state-of-practice, organizational aims, and challenges in adopting continuous software engineering practices. We compare state-of-practice with state-of-the-art to validate the best practices and identify relevant gaps for further investigation. Results: We found that companies start the CI/CD adoption by automating and streamlining the internal development process with clear and immediate benefits. However, upgrading customers to continuous deliveries is a major obstacle due to existing agreements and customer push-back. Renegotiating existing agreements comes with a risk of losing customers and disrupting the whole organization. Conclusions: We conclude that the benefits of CI/CD are overstated in literature without considering the contextual and domain complexities rendering some benefits infeasible. We identify the need to understand the customer and organizational perspectives further and understand the contextual requirements towards the CI/CD.

show abstract

“…It is then understandably if several researchers emphasize that the common problem in RT is time constraint or insufficient [8], [18]- [20]. Through various surveys, research in the RT field will continue to grow, with the increasingly diverse types of approach or a broader application domain, for more effective methods.…”

Section: Introductionmentioning

confidence: 99%

A Systematic Literature Review on Regression Test Case Prioritization

Rahmani¹,

Ahmad²,

Jalil³

et al. 2021

IJACSA

View full text Add to dashboard Cite

Test case prioritization (TCP) is deemed valid to improve testing efficiency, especially in regression testing, as retest all is costly. The TCP schedule the test case execution order to detect bugs faster. For such benefit, test case prioritization has been intensively studied. This paper reviews the development of TCP for regression testing with 48 papers from 2017 to 2020. In this paper, we present four critical surveys. First is the development of approaches and techniques in regression TCP studies, second is the identification of software under test (SUT) variations used in TCP studies, third is the trend of metrics used to measure the TCP studies effectiveness, and fourth is the stateof-the-art of requirements-based TCP. Furthermore, we discuss development opportunities and potential future directions on regression TCP. Our review provides evidence that TCP has increasing interests. We also discovered that requirement-based utilization would help to prepare test cases earlier to improve TCP effectiveness.

show abstract

On the use of usage patterns from telemetry data for test case prioritization

Cited by 14 publications

References 14 publications

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

A Taxonomy of Information Attributes for Test Case Prioritisation: Applicability, Machine Learning

Towards cost-benefit evaluation for continuous software engineering activities

A Systematic Literature Review on Regression Test Case Prioritization

Contact Info

Product

Resources

About