Improving the efficiency of functional verification based on test prioritization

Abstract

Functional verification has become the key bottleneck that delays time-to-market during the embedded system design process. And simulation-based verification is the mainstream practice in functional verification due to its flexibility and scalability. In practice, the success of the simulation-based verification highly depends on the quality of functional tests in use which is usually evaluated by coverage metrics. Since test prioritization can provide a way to simulate the more important tests which can improve the coverage metrics evidently earlier, we propose a test prioritization approach based on the clustering algorithm to obtain a high coverage level earlier in the simulation process. The k-means algorithm, which is one of the most popular clustering algorithms and usually used for the test prioritization, has some shortcomings which have an effect on the effectiveness of test prioritization. Thus we propose three enhanced k-means algorithms to overcome these shortcomings and improve the effectiveness of the test prioritization. Then the functional tests in the simulation environment can be ordered with the test prioritization based on the enhanced k-means algorithms. Finally, the more important tests, which can improve the coverage metrics evidently, can be selected and simulated early within the limited simulation time. Experimental results show that the enhanced k-means algorithms are more accurate and efficient than the standard k-means algorithm for the test prioritization, especially the third enhanced k-means algorithm. In comparison with simulating all the tests randomly, the more important tests, which are selected with the test prioritization based on the third enhanced k-means algorithm, achieve almost the same coverage metrics in a shorter time, which achieves a 90% simulation time saving.

Metric-Driven Verification Methodology with Regression Management

“…The limitations of current methods have been observed [3] and there are various approaches to mitigate them. Most of existing solutions is based on external solvers or libraries [10,12,16,17,21]. There are also other proposals that make use of other verification platforms, such as enhanced simulators [18], or testbenches developed using SystemC language [14].…”

“…But the discussed probability data is not directly known and needs to be measured. As an extension of [8], it has been proposed to prioritize some tests using their coverage contribution estimation [21]. Further improvements involve use of a machine learning and clustering techniques, in order to select tests in the suite with minimum overall execution cost [10,17], which corresponds to an increase of the verification efficiency.…”

“…-dynamic or static test constraints adjustment based on metrics data [4,7,12,14,18]; -avoiding execution of the stimuli that does not contribute to the coverage [13,16]; -selecting appropriate scenarios or seeds in order to minimize the executed test count [10,16,17,21].…”

A weighted resource discovery approach in grid computing

“…There are several model checker tools such as PAT (Liu et al, 2010), Spin (Holzmann, 1997) and SMV (McMillan, 1992). Model checking method is a significant technique to verify complex and concurrent systems (Clarke et al, 1999;Agirre et al, 2016;Podivinsky et al, 2017;Wang and Huang, 2016).…”

A symbolic model checking approach in formal verification of distributed systems

“…For improving these limitations, the symbolic model checking [35,36] with Binary Decision Diagram (BDD) has been presented by McMillan [34]. Some industrial tools such as NuSMV [37], PAT [38], Spin [39], and UPPAAL [40] are well-known for analyzing the system behavior correctness [41][42][43]. But, these tools have some limitations such as weak graphical user interface, the complexity of programming language and generating the automated temporal specification rules for verifying the system behavior [44][45][46][47].…”