Time-Aware and Weighted Fault Severity Based Metrics for Test Case Prioritization

Abstract

Test metrics succeed in analyzing the current level of maturity in testing and give a projection on the way to proceed with testing activities by allowing us to set goals and predict future trends. The objective of test metrics is to capture the planned and actual quantities: the effort, time and resources required to complete all the phases of development of the software project. Test case prioritization is an effective and practical technique in regression testing. It schedules test cases in order of precedence that increases their ability to meet some performance goals, such as code coverage, rate of fault detection. In this paper, we present a new metrics, based on varying requirement priorities, test case priorities, test case execution time, and fault severities. The case study illustrates that the rate of "units-of-test-case-priority-satisfied-per-unit-test-case-time" can be increased, and with improvement on testing quality and customer satisfaction. To assess the practicality of our approach, we apply it to a realistic example from the industrial projects. Also we summarize a test process measurement project of TECHZONE TM (Software development Concern with Testing Department) test teams, and analyze the effectiveness of set of metrics for cost, time, and quality to measure the quality of test process based on the results of the proposed metrics.

Functional Requirement-Based Test Case Prioritization in Regression Testing: A Systematic Literature Review

“…To address regression testing from specification modeling, Mary and Krishnamoorthi [45] have proposed a metricbased approach on the varying priorities of requirements, test cases, and execution time of test cases for prioritization of test cases. The main aim of proposing the "Average percentage of Severe Fault Detection per Time" (ASFD T ) metric [45] has been to evaluate the prioritized testing empirically, depending on the varying prioritization of requirements. Results of the proposed metric indicated that all other techniques (function coverage, statement coverage and requirement-based prioritization) outperformed the random ordering technique.…”

“…While FP and RV factors were not proposed for the TCP approach, that is why the results of the study cannot be generalized because more test cases prioritization factors exist in the literature. To prioritize test cases using test case execution time and the number of test cases is beneficial for the identification of maximum faults [45]. However, the challenge is the consumption of time on the execution of test cases which do not recover errors in a test suite.…”

“…Table 9 shows the types of faults/defects detected by test cases in subject programs, which are derived by the proposed TCP approaches in regression testing. Primary studies [9,44,45,58,90] did not mention number of test cases, Fig. 7 Issues identified in primary studies but revealed information about defects types.…”

“…From proposed TCP approaches in [46,48,53] test cases were used to detect the faults in source code of the systems. On the other hand, TCP approaches proposed in primary studies [9,45,52,56] used test cases to identify the functional defects of systems. TCP approaches proposed in [9,49] revealed requirements related risk defects of the systems under regression testing.…”

“…Researchers have validated their proposed TCP approaches regarding regression testing in various primary studies. It has been found that [9,[44][45][46][47][48][49][50][51][52][53][54][55][56]58] primary studies mentioned the names of SUTs. Researchers noticed that 'iTrust' system was widely examined in the primary studies [9,48].…”