Experimental evaluation of the tolerance for control‐flow test criteria

Abstract

For a given test criterion, the number of test-sets satisfying the criterion may be very large, with varying fault detection effectiveness. In recent work [29], the measure of variation in effectiveness of test criterion was defined as 'tolerance'. This paper presents an experimental evaluation of tolerance for control-flow test criteria. The experimental analysis is done by exhaustive test-set generation, wherever possible, for a given criteria which improves on earlier empirical studies that adopted analysis of some test-sets using random selection techniques. Four industrially used control-flow testing criteria, Condition Coverage (CC), Decision Condition Coverage (DCC), Full Predicate Coverage (FPC) and Modified Condition/Decision Coverage (MC/DC) have been analysed against four types of faults. A new test criteria, Reinforced Condition/Decision Coverage (RC/DC) [28], is also analysed and compared. The Boolean specifications considered were taken from a past research paper and also generated randomly. To ensure that it is the test-set property that influences the effectiveness and not the test-set size, the average test-set size was kept same for all the test criteria except RC/DC. A further analysis of variation in average effectiveness with respect to number of conditions in the decision was also undertaken. The empirical results show that the MC/DC criterion is more reliable and stable when compared to the other considered criteria. Though the number of test-cases is large in RC/DC testsets, no significant improvement in effectiveness and tolerance was observed over MC/DC.

On the relationship between two control-flow coverage criteria: all JJ-paths and MCDC

“…Recent work of particular interest in connection with MCDC is that by Kapoor and Bowen [6] who have empirically compared its fault-detection effectiveness against a number of other control-flow coverage criteria, but not 'all JJ-paths'. The same authors have also investigated analytically the fault-detecting properties of MCDC [7].…”

“…If each decision in a program (a) does not contain strongly coupled conditions; (b) forms part of a branching construct; and (c) contains only short-circuit 'and' Test ABCDE Result A B C D E MCDC TC 1 F----F 2,4,6 TC 2 TT--T T 1 8 3 TC 3 TT--F F 2 TC 4 TFT-T T 1 8 5 TC 5 TFT-F F 4 TC 6 TFFTT T 1 8 7 TC 7 TFFTF F 6 TC 8 TFFF- 6 1..7/9 0 0 0 0 0 0 1 0 ! # JJ 7 5..7/9 0 0 1 0 1 0 0 0 # # JJ 8 7..7/9 1 0 0 0 0 0 0 1 # # JJ 9 1..8/1 0 0 0 0 0 0 1 0 0 # # JJ 10 5..8/1 0 11 7..8/1 0 --------Infeasible JJ 12 9..10/exit 1 0 1 0 1 0 1 1 # # JJ 13 10..10/exit 0 1…”

“…In order to achieve complete MCDC coverage, one possibility, as can be seen from the penultimate row of Table 1, is to execute test set TS 1 comprising test cases TC 1 , TC 2 , TC 3 , TC 4 , TC 6 and TC 8 (i.e. omitting TC 5 and TC 7 ).…”

Coverage-Based Testing for Service Level Agreements

“…MCDC is a criterion that falls between condition/decision and multiple condition coverage [32]. This criterion has been shown to represent a good balance of test-set size and fault detecting ability simultaneously [34], [35]. MCDC is defined as a conjunction of the following requirements:…”

Full predicate coverage for testing SQL database queries

“…Modified Condition Decision Coverage (MCDC), defined in the RTCA/DO-178B standard [11], is a coverage criterion which has been demonstrated as representing a good balance of test-set size and fault detecting ability [12,13], studied in depth in the literature [14][15][16][17] and used for test suite reduction and prioritization [18]. MCDC requires that every condition in a logical decision has taken all possible outcomes at least once, and each condition has been shown to independently affect the decision s outcome.…”