A Methodological Framework for Evaluating Software Testing Techniques and Tools

Abstract

Such guidelines could be obtained by doing secondary studies on a body of evidence consisting of case studies evaluating and comparing testing techniques and tools. However, such a body of evidence is also lacking. In this paper, we will make a first step towards creating such body of evidence by defining a general methodological evaluation framework that can simplify the design of case studies for comparing software testing tools, and make the results more precise, reliable, and easy to compare. Using this framework, (1) software testing practitioners can more easily define case studies through an instantiation of the framework, (2) results can be better compared since they are all executed according to a similar design, (3) the gap in existing work on methodological evaluation frameworks will be narrowed, and (4) a body of evidence will be initiated. By means of validating the framework, we will present successful applications of this methodological framework to various case studies for evaluating testing tools in an industrial environment with real objects and real subjects.

Evaluating the FITTEST Automated Testing Tools: An Industrial Case Study

“…The design of the case study has been done according to the methodological framework for defining case studies presented in [13]. Although a one-subject case study will never provide general conclusions with statistical significance, the obtained results can be generalized to other similar software in similar testing environments of IBM Research [16], [8].…”

“…Finally, although finding faults in a reasonable amount of time is important, the amount of effort to set up and use the testing tools in the testing processes currently implanted should be important too. Hence, following [14], we focus on:…”

Virtual reality-based cognitive healthcare interventions: User-centered design considerations, guidelines and framework

“…It is worth mentioning that reliable techniques and tools for testing VR application software (Vos et al, 2012) in general ought to include: 1) a precise conceptualization of the application; 2) modelling of resources and system implementation; and 3) description of the evaluation process and how to carry it out.…”

Evaluating applicability of combinatorial testing in an industrial environment: a case study

“…Analyze the CTE tool For the purpose of evaluating applicability With respect to the effectiveness and efficiency of Sulake's current combinatorial testing practice From the viewpoint of the testing practicioner In the context of Sulake developing and testing departments Consequently, following [16] and focussing on the three measures of applicability 3 defined: efficiency, effectiveness, and subjective satisfaction. The research questions for each case study correspond to:…”